Is it normal that im having problems bonding with my boyfriends son?

I've been dating this guy for just over two years and I'm having problems fully connecting to his son. I have a ten year old daughter that goes to her dads every other weekend. She gets along well with my boyfriend, he's great with kids. My boyfriend has his son every weekend because his mother abandoned him when he was four. He just turned six years old in March. She has another child a 9 year old daughter (different father) whom she also abandoned both children live with her sister, their aunt. At first, my boyfriend had his son every other weekend but now that the mother isn't around he has him every weekend, and is even considering going for full custody. Honestly, its hard to bond with the child and mostly he seems angry at me all the time. He's either hot or cold. Sometimes really loving, other times completely cold. A couple weeks ago he actually said to his dad "Why aren't you looking for mommy? When she comes back ill live with her and we will move far away from you." His son basically started crying and my boyfriend looked heartbroken. He went onto explain to the child that his mother loves him and that he will always have his father in his life no matter what. His son went onto say that his dad is choosing me over him. I was devastated and felt terrible that the child blames me for his parents not being together. It bothered me that my boyfriend wouldn't say to his son.. that even if his mom was around we'd still be together, or something to that extent. He just reassured him that she's coming back and that she loves him. even saying that he loves the boys mother.. so disturbing to me. I understand that he's trying to help his son cope but I'm just terribly disturbed by this. Sometimes my boyfriend even has the nine year old sister over on weekends, who I really cant stand. Sorry to say! I feel terrible to even admit that but I'm really not a kid person to begin with..I love my own child but tend not to like other peoples children especially if they're not behaved. So I'm probably not the best person to become a step mother, especially to kids who are emotionally and psychologically abused. Not sure I have the capacity to take on broken children. I do love my boyfriend and enjoy our time together but is it time to move on? We speak of marriage and building a life together but this situation is so hectic, i'm not sure I want the extra stress in my life. He recently spoke about finding the mother of his child and trying to bring her back. He even asked me to help him. I wanted to walk away at that point. Not sure where to go from here.. I found myself thinking, am i being too selfish right now? What really is hard, is that we have these moments of connection (hugs and kisses) I do love his son...and I thought I was making a positive impact, but to see the boy so hurt broke my heart. I don't want to feel like I'm adding to the already existing pain of abandonment. That's why Im considering walking away. How do I help my boyfriend and his son through this without feeling like I'm doing more harm then helping? Or is it just time to just walk away? Thanks for your honest opinions!