Is it normal for an outsider to behave this way?

My husband and I found out almost 6 years ago that he fathered a child with an ex-girlfriend. His daughter was 7 months pregnant and needed a medical background history. We were both so excited to find out we had a daughter and a soon to be granddaughter.
After having the baby, the daughter started partying and doing drugs. We were only allowed to see the baby at the grandmothers convenience. Which was pretty much never. The grandmother at the time was living with her fiance. The two times the grandmother and her fiance were in our home, the grandmother stressed to us how odd she thought her fiance was, as did my stepdaughter.
We last seen our granddaughter when she was eight months old. We had been told that we could not take her for weekends or be alone with her because we were strangers.
A year and a half ago we were reunited with both of them. Our daughter had cleaned up and had become reunited herself with her daughter. They both moved in with us last July.
Our daughter for some reason or other feels that he should be a part of her daughters life. Every time he has been in our present, she stresses to us that he gives her the 'creeps' and she can't stand him.
My husband and I have been telling her that she does not need to let him take her daughter. We and several other members of her family share the same feelings for him.
The other day he called my stepdaughter and asked to talk to her daughter. Instead of clearing it with her mother fist he asked the granddaughter if she would like to meet him for dinner. It upset my stepdaughter that she was not asked before hand. They agreed to meet at a local diner. She dropped her daughter off and left. Later she picked her up and went home. I asked her why she left her alone with him? She replied, "I can't stand to look at him, he gives me the creeps."
I don't understand. Why would anyone leave their daughter alone with someone that they can't stand to be around themselves?
My daughter then informed me that when she was out using, her mother also had started using. This guy was left alone with our granddaughter.
I confronted him and asked why he did not inform the natural grandfather of what was happening? We were perfectly capable of taking the granddaughter and raising her. He informed me that he was there from the beginning, not us. I told him that he had no right to do that and make that decision. I asked why?
He has been so obsessed with this little girl. He sends DVDs to her of when she was a baby and he's the one that is mostly focused upon when she was a little girl. It seems as though he wants to rub it in that he was the only one in her life.
There is also an incident that happened when he had her. Apparently there was blood in her panties. The state investigated and supposedly it was a 'bathtub' incident.
When I confronted him, I told him it was time to move on. I didn't feel that his behavior was appropriate. I also shared with him how family members felt about him giving them the 'creeps.'
I feel as though this relationship has gone too far. I have sat back and not said a word because I'm just the step mom. I couldn't sit back any longer. Too many red flags and too many of my stepdaughters own family saying it needs to stop. I figured if I didn't say something I would be guilty of blame for something bad happening. Now I'm the bad one. I only said what I had heard from the others. Including my husband. I don't fear this guy and I feel that it had to be said. I told him that he was nobody. No relative. No member of her family. Nothing.
Should I apologize? Make amends? Did I make a big deal out of nothing?
I have a lot of emotions because of the fact we were not allowed in our daughters life. I know that if we had been her life would have been so different. I want to be there for our granddaughter to be her voice and give her a better life than her mother was given.