Boyfriend has trouble getting it up/ staying hard

My boyfriend (who is 27) and I met over a year ago whilst he was on holiday in my city. We really hit it off but nothing other than kissing happened when he was here the first time. He then came back to see me and when we tried to have sex he lost his erection (he said it was due to the condom, which I thought was fair enough). He kept coming to see me and we never had sex, although we tried every time and he did get an erection and cum from BJs and HJs so I thought it was still about the condom.

He asked me if we could try making it a serious relationship and I agreed, even though we lived in different countries, I was really in love with him. After that I went on the pill and I was excited that we would finally be able to have sex without the condom being an issue... except that when I went to visit him he couldn't stay hard for sex. I was a little confused, this has never happened to me before, but he's a shy guy with not much experience so I didn't say anything. 4 months went by until I brought up the fact that we never had sex (we did do other stuff) and he said it was because we lived in different countries and when we saw each other he felt too under pressure for everything to be perfect to perform. I guess I understood but now I was getting so frustrated! And wondering why was it only actual intercourse that was a problem!

We went on holiday together and had the most amazing sex for 10 days, several times a day and I was so relieved! But then when we went back to just visiting each other it kept happening. By this point we had agreed to move to the same country and live together and he told me once we had a routine and we lived together it wouldn't happen anymore. 

We've been living together for a few months now and it's still a problem!! It's like, he's hard and he wants to have sex but as soon as he tries to put it in it just goes soft and then he tells me I need to play with him more but if I do that I basically just get him to the point where as soon as he's inside me he's cumming! My problem is that now I'm getting so worried about it that sometimes I'm too tense for him to even get inside... so it's a vicious circle!

I love him so much but sex is so important to me that I don't know if I can realistically see a future with him. He tells me it will get better, he says it's all because it happened once and now he's worried it will happen again, but it's been over a year now... how much longer should I give it? I know it's difficult for him but it's also causing me to have self-esteem issues and I often feel depressed and I can't talk to anyone about it. I know he loves me and he tells me he finds me attractive... but surely if that was the case then this wouldn't still be a problem! Before I was with him I used to have regular sex and I miss it. I have been so patient but now I am really starting to lose it. Can anyone share any thoughts or experiences? Or give me any advice? I feel like I've tried everything including dressing up in lingerie etc but trying too hard definitely makes it worse!

Just FYI he is healthy, doesn't smoke or use drugs and doesn't drink very often. I'm sorry this post is so long... i guess I just need to vent!