Is it ok for me to cut ties with my grandfather?

I'm a female, 25 years old.
My family has lived abroad for 15 years, so I didn't get to meet my maternal grandparents except on some holidays, we returned to live in the same country when I was 23 (2 years ago), I never noticed before that my grandfather didn't like me, because I didn't care much before, but I've seen it clearly since we moved back.

He is a mean person, not loving, and he's especially mean to my mother, not directly but it's obvious she's not his favourite (he has 8 children) but she's ok with it.

At first I thought that he's just not used to us, but as time passed I realized that he has special relationships with all his grandchildren, I'm the second grandchild in the family of about 30 grandchildren, and I'm the first girl.

I thought that maybe since my mother is not his favourite and he doesn't like my father then he just hates our family, but he's good to all my siblings and extremely cold towards me.

I'm a really warm person by nature and always try to help everyone around me, I try to always be nice, but I've stopped trying with him because he made me cry on more than one occasion.

My grandmother loves me, as I'm one of her favorites, and my paternal grandparents love me too, it's just him.

He usually loves beauty so I thought maybe I'm too ugly for him to love.
Or maybe I'd done something as a child that I don't remember.

I don't know if it's normal for him to act so cold,and if it's ok to cut ties with him.
Although I'm not sure how I can do this since we're a very close family that meets every weekend, and I love my grandmother so I don't want her to get sad, but he's really affecting my mental health.