No, I mean if your spouse only cares to keep you away from other women, but doesn't care which men you associate with.

Ah, I see what you mean. I can't really speak to that from my own experience, which was that of a gay man married to a woman who set great store in the idea that she had turned him straight. (She had not, of course. She merely found him too young to have realized that what one's body says at any given time need not be taken as the final word on anything, and went to a great deal of effort to deny him that realization.)

Perhaps a place to start might be: What would we say of a man who only cares to keep his wife away from other men?

I'd say pretty much the same thing. I'd only consider that man controlling/isolating if he kept her from the women too.

I'm not sure I understand how it is acceptable for one's intimate partner to forbid on pain of punishment that one have contact with anyone.