Relationships...

Phew, Right i have been married for 20 years, i have 3 children, one of 18 one 16 and one 15.
I have been fond of this guy at work, he is 31 for ages. He is a confident and very helpful, kind, caring and giving if very slightly selfish but very self assured man who has a family and two children.
We started to joke about sex around 2 years ago. I asked him to do something and he said "blow me" my reply, for some reason was don't say that because i will. I don't know why i said it, i have always had a softer side for some guys, who i find attractive (as people, not just good looks) but this was at work, in a factory!!! We joked around and we were friendly for well over a year when we were alone, which was frequently at work. Over the next few months we became closer still. He kept looking at my crotch and he gave me a shoulder massage. a few weeks passed whilst he did this every other day then and he his hand under my shirt, on my breast and caressed it. He started to toy with my nipple. I put my hand over his mouth and kissed my hand, we were very close. This carried on at work several times a day in some instances for a long time, months in fact....
Eventually he undid his pants and i slid my hand inside feeling him, god it was fantastic, there really is no other explanation, i loved it. He got very excited very soon and i masturbated him, as it were... This was became the norm after a few weeks. The weeks passed and turned into months....
So more recently i have given oral sex several times and again i liked it. I cannot explain why but i at least need to explain it to me. I really need to because i feel i am bound on a crash course, nothing nasty but {i know, i g*d damn really know that} i really should know better at my age but could this have been in me and trying to surface in 1980 when i was 17.
Wife? yes and children we both have.
Confused and wondering what to do, we both know we cant walk out and set up home, that will never happen so what can we do.
Please don't make fun or anything, this is not just an issue, it's at least a dozen peoples lives, why didn't i think of that two years ago? i did, but the urge was so great......