Stupid fantasies

i fantasize about all my older female work colleagues. and they are all older than me like in their thirties and forties. i have sex with my wife but think of my older colleagues at work... is that bad???