Should i really give a f#@^?

Ok, I work at a resturant. Which is a very social place with both staff and customers alike. I can honestly say I handle serving costumers, as well as preparing food well. I have never been faulted for laziness or irresponsiblity. My problem Is with social interaction with some co-workers. A chosen few, that seem to tell me stories of past relationships, Stupid ex's, and not having enough hours at work. When Im approached, I try my best to give them the answer that will miniumize the interactions. They say Im anti social, or even stuck up. But really I just don't care. I Do not interact with them outside of work. none of them control my payroll. Really, If they never said anything to me all day, I would have no problem. Half of them complain about not recieving hours, but call out sick 3 times out of the week! I really dont feel like I should have to be friends with my co workers. Just professional. I never once said anything out the way. I never really talked much. Im like this way with everyone.I have some social anxiety issues.But it's never gotten in the way of my work. I need the money. I don't need the childish co workers. Am I wrong for wanting to just do my work and go?