Is it normal that I often find poor people annoying?

Totally agree. Obviously, like you said, this doesn't apply to all ppl, but for a great many it's absolutely true and really messed up