People comitted suicide from dealing with us even when doing so made no sense; they simply let their ignorant fears of the Big Bad put them in a bad place, mentally.
When a parade of kooks and idiots testified to Congress in 1998 that we were all baby-eating monsters, NO ONE stood up for us. Horrific legislation that left the agency permanently hamstrung resulted.
I don't know how you missed this while working there, but the IRS deliberately cultivates that reputation. They WANT to be known as baby-eating killers, they want people to fear dealing with them so much that they don't even risk anything which could result in an audit even if it's 100% legal. The IRS has been doing government by terrorism for a very long time now, and it's quite effective.
Every so often the people get uppity, so the IRS has to pull something like holding day care students hostage until the parents pay the school's taxes. That usually works.