He couldn't go for a second round after orgasm, is it my fault?

I was seeing a man last year and then things eventually cooled off, but have recently started up again. Last year when we had been together he always seemed very pleased with the sex that we were having and even said I'm the best he's ever had as far as oral is concerned. He almost always ejaculates very quickly and says "now you need to be a little patient" and later we'd be able to enjoy a second round that lasted a lot longer. Well after a year of not seeing eachother we hooked up again. This last time though I tried everything to excite him for the second round but it was impossible, I almost had the impression that I was maybe bothering him in the end but he stayed we talked,cuddled and caressed for a good hour and a half after the first quick ejaculation, instead of getting dressed and running home (to his wife). Could it be that I just don't turn him on anymore? He's an alpine guide and had climbed that day, could it have been exhaustion? Should I take it personal? We have a habit of writing very steamy text messages to eachother and that is always exciting. Please send some good honest advice. BTW, I am also married but obviously not happily.