It depends on the type of surgery you've had, but you shouldn't go swimming until
Generally, after your stitches have been removed or have dissolved and your wound has fully healed, you should be able to swim in the sea or a swimming pool. Once a wound has healed, the risk of infection decreases.
Stitches can usually be removed within 7 to 10 days of surgery, although it depends on the type of wound. Absorbable stitches can take longer to dissolve.
You should avoid swimming for longer if you have another condition that increases your risk of infection or delays healing. You shouldn't swim if you have open wounds.
I think you should follow the advice of your doctor and don't go for swimming because you need to take some rest when you fully fit then you can...
So wouldn't be wise?