Recording a conversation without the consent of the other party even for the purpose of providing evidence requires a warrant, under the first amendment and the laws governing free speech. While I understand the intentions and agree that attempting to resolve it by providing clear evidence is reasonable, the simple truth is that under US law recording conversations is prohibited without the oversight of a judge who can determine whether or not it is an appropriate exception to the right of free speech.
You're making it sound like all-party consent laws exist throughout the entire US. Only twelve states require all-party consent: California, Connecticut, Delaware, Florida, Illinois, Maryland, Massachusetts, Nevada, New Hampshire, Pennsylvania, Vermont, and Washington. In the other 38 it's perfectly legal to record the conversation as long as you are a party to it. This isn't anywhere close to a first amendment or free speech issue. Your right to speak freely does not imply a right to make others forget what you said or prevent them from testifying about it—and the only relevant difference between a recording made in person and one's own memory is that the recording is a more reliable form of evidence, which is all to the good.
It's the all-party consent states, like this one, which are being unreasonable here. So long as the person speaking is aware that you can hear them, they have no reasonable expectation of privacy from you and you ought to be perfectly free to record what they say.