I avoid my neighbors, even though there's not a rational reason for it

I have had a variety of neighbors/roommates- Old/sick/crazy/friendly/bitchy/young/smart/totally stupid/etc...Some have been awful and there was no reason to want to be around them, for me at least. Others have been friendly and have also wanted me to hang with them. Regardless of the situation, unless I was already good friends with my neighbors/roommates before they became neighbors or roommates, I do not want to spend much time with them. Sure, the idea of having a 'friendly neighbor' is a nice idea, but when it comes to actually having a friendly neighbor, and being one myself, I'd rather just say no. (Of course, being a friendly roommate involves more energy than with neighbors, and I tried as best as I could to get along with roommates, without saying yes to most of their invitations)...This is not to say that I am not polite, but I can not bring myself to hang out with my current next door neighbors, even though they are my age and aren't completely weird in a bad way and have at least a tiny bit in common with me, and also want me to come over. I avoid them I think because I like space. If i start saying 'yes' to their invitations and their drop-ins, how do I draw the line? I like space and privacy. I guess those are the things I'm trying to protect. Is this a normal behavior on my part?