Is it normal that i distance myself from anyone i have sexual relations with?

I'm not antisocial or a social butterfly. I'm mostly average, I have one close friend and the rest are more like associates. One problem I've come into is that once I have any sort of sexual relations with someone, I pull away. I realize that I'm doing it and I know that sex is the reason but I can't seem to stop myself. I'll be really happy that it happened for maybe a day, but when I have to face them or acknowledge that it did, I become extremely uncomfortable. I won't even like the person anymore. It worries me because I'm completely sure and comfortable with it when it happens, but later, I just want to forget about it.