Is it normal for a husband to be close friends with 20 year old neighb

So here's a little background on my home situation. I am 26 and is my SO. I have been with my SO for almost 8 years. We have a 4 year old daughter he drives trucks for a living and we only see him about 6 days out of the month with a few nights here and there if he takes his rest nearby. We currently rent our home and have been there for a little over a year. As to be expected we are "cool" with our neighbors. It is a dad and his 2 sons one is 17 the other is 20. Okay here is my situation.

Neither of the neighbors boys ever really partied and we kept our relationship with them very neighborly. Although the 20 year old, Joe, and i have had a couple tiffs bc he is a taker and very self absorbed so he would try to take advantage a lot. Joe recently got a GF who is an ex meth addict but ever since she got off the meth she replaced it with drinking and now Joe is drinking a lot too. She is 18. Which to each is own. I was that age I know how itis and its none of my business. Well she is living there now. So anyway, recently Daniel, my husband, has been hanging out with them and drinking. Which I am uncomfortable bc I don't care to be around them bc I'm just at a different place in my life then they are. I have responsibilities, I'm a mom you know etc. but now it's as if they are all best friends. And he is in a way neglecting us. Not to the fullest extent. It's kind of hard to explain over th computer. But basically they ask Daniel for some of his liquor to make drinks and always want to sit and BS with him at our fire pit and just constantly are coming over. Now my husband is in no way an alcoholic but he likes to sit out by the fire and just BS. Which I love to do as we'll. but not if the neighbors are CONSTANTLY over so I stay inside. I don't want to but I feel it's ridiculous that I can't even just enjoy a night at my fire pit without being stalked. I have said something a few times but I am sick of being the bad guy. Isn't this a problem? I strongly feel as if this is not an appropriate relationship to have with 20 and 18 year old neighbors and You can't always just hang out. You have a family and a wife and your basically choosing these other people over me. I have spoken with him about it and his trucking career plays a huge part in it and is a big stressor but that doesn't mean you can just throw all your responsibilities out the window and act like a kid again. Am I right?

Well last night Daniel tells me his is running to the store to get stuff to change his oil. When he got back my daughter and I went to check on him in the garage and the 18 year old GF is sitting there. Okay annoying but whatever so my daughter and I went back inside. Well I later found from someone NOT MY HUSBAND that this girl went to Walmart with him. Okay let my clarify that in no way do I think anything is going on between them. That is not ,y intentions with this post i know he would NEVER do anything but is this not inappropriate? I talked to him about it last night and he says to me that he doesn't see the big deal that they are friends. I do not think that a 26 year old should be hanging out and drinking with an 18 year old and 20 year old. Not only is it illegal but it is not age appropriate in my eyes. He says he just wants to do what he wants when he is home. HELLO YOU HAVE A FAMILY! What about us? What about me who only gets to see him 6 days out of the month. Please give me feedback how to approach this situation in a way he will understand I'm at my wits end.