Is it normal to need to "recharge" your tolerance for certain people?

I am not an extremely social person to begin with. I can make a small group of friends in different circles or classes (I'm in college now), but I'm not someone who has 500 friends and is always out partying. For some reason, no matter how much I enjoy one person or one group's company, if they hang around me way too much or I hang around them way too much, in a flip of the switch I feel like I'm being drained by being around them. The first few times I began to feel drained around people, I would never speak to them again and have not spoken to them again and that was the end of it when I was a kid.

As I got a little older, I would feel these feelings again about people that just days ago I thought were so fun to be around, but I would just "fight through it" until I got to a point with one particular group that I began to feel dread all throughout the day that they would want to talk to me, and then I was annoyed that they saw me and wanted me to hang out with them. Once I was fed up with the feeling that I was being drained by being around them (I still felt normal around people or groups that I didn't see all of the time) I cut off contact as much as I could and I never could give them an explanation as to why I was avoiding them (they hadn't done anything wrong). After a couple of months went by, one of the friends in that group demanded that I stop being an asshole and avoiding them and I did chill with them again. I realized this dread had went away and I did not feel like I was being drained to be around them. When asked about the situation, the only answer I could give was "I don't know" because I didn't know how or why I felt the way I did.

It's not just male friends that I can tire of. If a female friend talks to me too much or I talk to them too much, I get to a point where I just stop saying anything or responding for a while which makes me think I could never get married to a woman because it would be strange to just move out for a month everytime I needed to "recharge".

TL;DR: Does anyone else need to recharge their tolerance for people that they are around too much?