Am I a bad person for being like this?

the human animal is a beautiful and terrible creature, capable of limitless compassion and unfathomable cruelty. If you wish to find that which becomes the dividing line betwenn mankind and other biological classifications, it rests not in brain size, dominance, or even emotional capability, but rather in the unique capacity for human beings to reflect on their actions, and show regret; It is most certainly their ability to empathize that gives them their position. All animals understand love and affection, but only man shows the propensity to place himself into the shoes of another life form. losing that capability, among individuals of the species, reduces them from their much heralded position and readies the climate for the likely fall of man; The fall from grace.

If you see it as a matter of "walking on eggshells to tolerate someone" then I really don't even know what to say to you. It shouldn't be an effort to tolerate someone who's essentially a good person outside of being disabled. Don't bend over backwards to them, but don't be a dick either. You can hate someone for being an asshole, you can dislike someone for doing something fucked up. But to judge someone based on the circumstances of their birth? Seems to me like you're forgetting your humanity. The world would be a much better place if we all just gave a shit about each other a little bit more.