I'm not sure how old you both are, it does matter for a response.
I will tell that no matter your ages, trust issues will destroy a relationship.
If you're talking about your future together, as in marriage, then maybe he's scared. Scared of being tied down, having responsibilities, becoming a dad, all that. He might not want to "grow up" and have to stop hanging with his friends.
It does sound as though he's jealous.
I know you've invested three years with him but people DO change, especially when they're young. Maybe it isn't meant to be.
You've both cheated in the past so this hasn't been a perfect relationship.
Let it go. Maybe he'll realize what he's got. Maybe you'll find someone else that does trust you.