Is it normal to hate when a guy im not with calls me baby?

Yes and no.

If you're "hooking up" with him then ... meh.

but if he's a total stranger yeah that would suck