Is it normal to want other men's wives more than other women?

You pretty much answered this yourself. Most people are aroused by the concept of forbidden fruit.