My wife has been on birth control most of our time together so we rarely used condoms, usually only when her birth control might be compromised due to medication or other things. After the vasectomy I had three sperm count checks as I needed two checks in a row with a zero count, so took three checks to get there.
My wife currently has an IUD but that is due to be removed in the next 18 months and then she shouldn't need anything else as I've been "fixed".
I agree sex with a condom can be less than desirable, but if you are concerned see your doctor and get the sperm count done, if you are shooting blanks then you should be good to go, but talk to your doctor if you have any concerns.