Boyfriend hardly says yes or no!?

My boyfriend almost never answers any question with a yes or no answer. He usually just says "I don't care" or it doesn't matter. For example I'll ask him if he wants to go for a walk or to the movies or something and he'll just say I don't care. It makes me think that he really just doesn't want to but just won't say it. I am always the one suggesting places when we go out bc he hardly ever will, he knows that I'm open to his suggestions bc anytime he has suggested anything I've been game.

He does this when we are going out to eat too. I'll suggest somewhere and he'll just be like it doesn't matter. But then if we're arguing about something totally irrelevant, he'll turn around and say that I never wanna go to the places that he wants to like restaraunts when he's the one that never says yes or no or even suggests anything!

I understand that yes or no is not always needed, but like I said he does this all the time and it's really starting to annoy me, especially when he trys to turn it around on me as I stated above. Also I don't want to have to make all the desicions when it comes to everything, I feel like in a relationship it should be 50/50. I've tried talking to him about when he does it but he seems to think there's nothing wrong with it.

Is it normal for people to be like that? And is it normal for this to bug me so much?