It's convenient to complain about the IRS, but its flaws are a result of our own animus. Note the flaws of the agency are separate from those of the underlying tax code it has to administer, which it does not write (blame Congress for that).
We don't want to fund the IRS, so its budget keeps getting cut, while the list of demands placed upon it increases. Nobody likes the IRS, so it has difficulty attracting high-quality job applicants. Would you want to work for an agency constantly being berated for doing its job? The workers are forced to do without simple benefits private sector workers take for granted, such as free water coolers and coffee because of public stinginess. I recently read an article in a trade publication that states the IRS has fewer than 750 workers younger than 25 out of a workforce of almost 70,000. The figures aren't great for under 35s either. With that kind of recruitment, it's little wonder that they are a bit behind the times.
Of course, there are the scandals, but those have involved small subsets within the organization. If one subgroup of 5 employees in Exempt Organizations did something wrong, public opinion pillories the remaining 69,995 employees. One example of waste becomes an assumption that everything is waste.
To share a personal story as a tax professional: I applied to the IRS coming out of school out of an interest in protecting the public interest. The pay was just over 1/3 of what I was being offered in the private sector (albeit with slightly better benefits). The recruiters did not exactly exude excitement about their jobs. Ultimately, that was too tough of a pill to swallow. Now, I help companies minimize their corporate taxes.