There is NO way for your doctor to know EXACTLY when you conceived.
Conception IS NOT immediate. It can take upwards of 5 days.
You would likely have ovulated sometime between March 21 and 26th. Sperm can surve about 5 days. The egg is only viable about 48 hours after ovulation.
Having sex on the 18th COULD get you pregnant, then again, no form of protection is 100% effective either for when you had sex on the 21st.
Odds favor it being the ex, just because of protection with your bf on the 21st.
You'll need a DNA test to know for sure.
Good luck. Hope it helps.