I think he told you that because he wants to be exclusive, and through your vagueness and refusal to answer questions, you've basically told him no. And so he is passive-aggressively showing you that he doesn't like it. Kind of saying well if you don't want to be exclusive then I can see other women and not hide it so this is how I'm gonna act. And if you mention it he will say well we're not exclusive so you can't complain.
see the review