Is it normal to think someone loves you even if they say they don't?

Ok, I have known this guy for four years. We had a fling. It was good. He would not admit to having any feelings for me; he said it was all about sex. Yet we had the most amazing time together and were very much on the same page emotionally. We laughed. We cried. We shared deep secrets.

He would do things like rub my face saying that he just wanted to know if his skin was rough from working. He joked a bunch of times about me asking him to be his girlfriend, he even once said that I better be faithful to him, but then he said he was joking.

Long story short I ended up having to leave the area and met another guy who I eventually moved in with. Before I left I asked guy A if he wanted to be with me and he said no so I left him alone. Months later I could not stop thinking about him and called him. He sounded really excited and pulled off the road to talk to me. He expressed being upset that I had vanished and said I could have been with him the whole time. When I asked him what that meant he said it meant nothing.

We have talked many times but never gotten together. He has a girl and I am still with the other guy but he is very emotionally abusive so naturally I think about guy A quite a bit sometimes. Every time we have talked about getting together, I chicken out and don't go. I am afraid of what I would do and don't want to cheat.

He still maintains that it's all about sex. But he texts me and says stuff like "I can't stand it anymore, I need you," and "I want to make love to you all night" and "you need to get your shit straight." I asked him recently if he had feelings for me and he answered, "What difference does it make?"

I am so confused! I tell him I will not have sex with him under these circumstances and he says ok, goodbye but then every month or so he is back wanting to talk to me and see me!

It may not matter but he was sexually abused as a kid pretty badly and it seems to me like he might be using sex as a way to connect (different than normal).

He called me early one morning this week and then later denied calling me and said it was his phone! Lol

He won't go away and it's been 2 1/2 years since I've seen him. Do people who really just want you for sex still call you after all that time? What is his deal? Does he love me or have feelings for me?