Is it normal to feel constantly let down?

I met a guy who seemed perfect for me. The first problem is we live a few bus rides(about an hour)away from each other. I tried not to let it bother me as he expressed very quickly that he felt the same as me and things were going really well. The first let down was after seeing each other for a while he was still quick to correct any one who mistook us for being a couple and insure them he was not my boyfriend. I got really frustrated and paranoid about this because he was telling me he loved me and talking about our future together and seemed very happy. I eventually brought it up and said if you don't want to be with me it has to end because this is a head fuck.. He said 'of course I want to be with you you are my girlfriend I suppose I'm just scared.' I tried to put my paranoid feelings aside and remind myself he's nothing like my previous boyfriend who cheated and lied about it twice.. Its different and not fair to judge him on my own past. We have been together a while now and have a lot of great times when we are together but I feel constantly let down by him. He lost his job so has a lot of free time and isn't too bothered about finding a new one. I work 4 days a week and spent most of my days off going to stay with him, he rarely visits me as he says he has no money, but since he's living with his parents the amount of money we both have to live off (after I have paid rent and bills)is pretty much the same and he doesn't have to buy food for himself. I feel pathetic for it but really resent that that's his excuse for not coming to see me. He will often say he'll call later or he will come to see me and then forgets or calls late to say he will leave soon and come over then arrives 3 or 4 hours later by which time I want to go to bed. He will constantly remind me how poor he is like its a permanent excuse for him not having to do anything Id like to do or him coming to see me and I feel suspicious and resentful all the time. I hate feeling like this, I know he loves me but he spends all his money on alcohol and drugs and wont save just enough to catch the bus to come and see me once or twice a week. I'm missing out on things I want to do, even on seeing my friends and family because I'm going to see him all the time, when he can come to see me on an evening after work but never will. I feel too tired after work to make an hour journey there and back or have to get up 2 hours earlier the next day to get to work in time, but if I want to see him this is what he expects me to do! He could come to mine then we get to spend all evening together and he can get up and leave mine and do what he likes when he likes the next day, but he wont. I'm always pissed at him but then I'm always missing him too, I feel like he just doesn't have the time for me and I'm the bottom of his list of priorities and I don't know if its just me expecting too much or we're just not as perfect for each other as we first thought?