I can understand conservative's (Tea Party types) dislike - it was pretty damning the selective enforcement of tax laws against conservative organizations
Wrong. This is one of the most widely debunked myths in modern politics. The IRS was investigating a wide range of new 501(c)(4) organizations that cropped up following the formation of the tea-party movement. 501(c)(4) organizations are tax-exempt nonprofits, and as such they are obligated to avoid being used for political purposes. Political organizations are supposed to be taxed. So of course, like any responsible agency should, the IRS took time to carefully examine the sudden influx of obvious right-wing political organizations seeking tax-exempt status. Even so, not a single one of those organizations was ever denied 501(c)(4) status.
At the same time, the IRS was giving similar scrutiny toward more liberal leaning organizations. They went through the same process as the rightwinger political organizations, but unlike the rightwingers, the IRS decided to go ahead and deny status to a number of seemingly-liberal organizations. Of course, this didn't particularly matter to the rightwing media when the story broke. They proceeded to lie, scream incoherently and flood the airwaves with manufactured outrage. It was part of the ongoing effort by the right to paint themselves as victims, and they still won't shut up about it no matter how often it's been debunked. As we've seen again and again, if the evidence for wrongdoing never surfaces, the right will just turn a non-scandal into a conspiracy theory. See: Benghazi, Obama's birth certificates, etc.