I'm attracted to my cousin and want to be in a relationship with him?

Okay so, it's been happening since I can barely remember. He is my father's brother's wife's brother's son - I know we are not related by blood or anything but still he has been visiting our family ever since we were kids. He is a year older than me and we got along quite well and there was a point when we used to call each other best friends and stuffs we never had that "cousin" feeling but our families sure thought differently.

Alright so, it was just after summer when we were in 10th grade and that's when he told me how he really felt about me and that he had loved me ever since he can remember. I was quite surprised but when I thought about it the idea didn't seem bad and we went out for about a few weeks or so and later I got paranoid and told him that it could never work and I was just "trying things out", which totally, totally, broke his heart. He didn't speak to me for a while and then after some months we got in contact again. He tried his best to keep things normal with me and at some point, things did get pretty normal. He had his share of girlfriends that he made sure to introduce me to but none of them quite worked out well(I was not involved in any of these issues or that I was ever jealous or anything). But he cried a lot and came to me after his break ups and literally told me about everything that happened which got quite awkward at times but he was really broken after almost all his break ups.

The main "problem" surfaced after he got into university and he moved to a pretty far place. I still stay in our hometown and now we barely talk. I think him and I both got over what happened a couple of years back but he got quite distant and barely talks to me or calls or ever reply to my texts and to be bluntly honest, I kind of miss him. Heck, I miss him a lot and suddenly he is too secretive even if we do talk at times, it's like he's forcing himself to talk to me? We have had been close and all of this just doesn't feel right to me and I've realized that I do like him but just last winter he was home on vacation and visiting us and then us kids were just messing around and playing this game and he asked me, "did you ever like me or do you" and I'm sure it was strange because everyone was in the room and we were friends and everyone was around so I just covered up saying, "Of course, I like you." as in a total friends tone and I think he got upset and left the room. Before leaving that time I went to see him off at the airport and he said "you know I didn't mean it in that way." and asked what my reply was going to be and I just said, "I thought both of us got over that... a long time ago." and he left.

But it's not because what I said that he's avoiding me. I have been like that almost all of my life and he knows me better than anyone else. I just feel like he doesn't need me anymore and that he is happy with his new life. But the problem is I can't really tell him what I feel about him since he's a part of the family and always was and my father's relations with his family is not best only him and his siblings come by for our sake. Also, none of our family would ever accept this because to them we are "cousins" that is the reason why I never continued this in the first place. Also, he had girlfriends and I saw him after his break ups and that is not how someone looks like if they say they love you so sometimes it's hard to believe him. I mean I do want us to be a thing but at the same time I'll be okay if we continued as friends(except the fact I started having these fantasies not long ago...) but I don't want him to act like he has been.. avoiding me. I really don't know what to do anymore.