Is it normal that my cat acts this way whenever I do this?

Some breeds are not as affectionate as others. This can further branch into each individual cat. If your cat doesn't like it, try not to do it so much.