Is bastard a swear word in england/throughout the uk?

So my aunt just arrived in England yesterday with her 11 year old son. While they were on the plane, there was a man sat next to them who was extremely rude. The man was complaining all the way to England about how Kiwis are superior to all and that he shouldn't even be visiting England because he's letting New Zealand down.

Anyway, after they arrived in England and went to this tea shop close to their hotel, her son and her were talking about how rude the man on the plane was and her son called him a bastard twice. The first time he used the word bastard, a man who was sat at the table next to them turned his head to stare at them and he shook his head. Then when her son said it the second time, this woman sat at the table behind her nudged her on the shoulder and said, "You need to control your son" and my Aunt was confused so she said what do you mean and then the woman said something like, "Your son is swearing and he's just a child, I don't think it's appropriate" and then the man agreed and said something like, "Yeah I was shocked when I heard the "b-a-s-t-a-r-d" word come out of his mouth" and the woman nodded and said "It just bothers me when a Mother lets her son swear like that". She then said I don't understand and the woman said "you're in a public place, if you don't understand how saying bastard in a public place is wrong, I just don't know how to help you out"

This word is so harmless, I don't even get how it could be a swear word. I hear three year olds say it, it's the same thing as saying, "the man was an idiot" or "the man was bad" lol. These people were shocked and I'm so confused. My aunt laughed so hard when she told us this story as she never swears nor does her son and for them to say they were swearing was crazy odd.

So is this a swear word in England/uk?