More problematic is that vote by mail isn't private / anonymous - your actual vote is visible for all to read.
What kind of system are you using? Typically you don't just write your candidate's name on the back of a postcard. The ballot itself does not contain any voter identification. You seal the completed ballot inside a blank envelope, and then put that envelope inside another envelope labeled with your voter identification. You sign the outer envelope and mail it in. Later someone checks the identification and signature, confirms that you didn't already vote by mail or in person, and removes the sealed inner envelope—they see who voted, but not the actual votes. The outer envelopes are discarded, while the anonymous inner envelopes containing the ballots are mixed together and opened by someone else. This second person can count the votes but does not have access to the identities of the voters.
The system honestly isn't all that bad, though it does have certain vulnerabilities. Most notably, someone could watch you fill out and mail the ballot and also ensure that you are unable to reach the polls to override the mailed-in responses by voting in person.
Of course, all this effort is pointless if one can simply ask voters about their selections and reasonably expect a truthful answer. The practicality of the secret ballot fundamentally depends on voters' willingness to lie, which makes it rather dubious, morally speaking. The fact they they could lie and that there is (ideally) no way to prove that any given response is accurate means little unless a significant fraction of individuals are actually willing to defraud prospective vote-buyers, which would reflect poorly on their character and is not really a form of behavior which ought to be encouraged.