IIN if I want to break up with him because he thinks I'm a bitch?

You can't let a silly thing like that stop you from doing what you want. Who cares what your ex has to say... he's your ex, they all talk shit.