I have pain during intercourse, but just with one guy

I suffer from this pain after having sex with this one guy. I have sex with others and I don't have this problem? Is it that the guy is too big in general or something?