Is it normal that I often find poor people annoying?

My rich brother once said whilst driving past a lesser well off area. "Don't look at them they're poor'.
Hmm, this must be how others perceive my situation too, very sad.