Is it bad for me to think he's selfish??

So I've realized I'm slightly unhappy in the relationship I'm in but the thing is I really love the guy and I want to find any way to make it work. I seem to be the only one unsatisfied. My boyfriend has always been a sickly child and I've always been there for him. I don't usually complain but the other day I tried to with him about my little cousin and how I have to babysit her for the week and how she's getting on my nerves because she's so spoiled and has attitude problems. He got smart with me and told me it was my decision and after I told him it wasn't, he told me that I needed to deal with it then. The next day he acted like it never happened and when I asked about it, he just said he was stressed. I don't tell him to deal with it when he's sick like everyone else tells him. I kind of think maybe he takes advantage of that. I feel like he's not even interested in what I have to talk about, ever. All he talks about is how he doesn't feel well and it is bad of me to think of him as selfish?