Because the more people who are under "oaths of silence" that have your records, the less like "silence" it becomes.
And yes, if you can assume that your government workers are only acting in the interests of their job, and never overreach, and never act politically, you might well be justified in allowing them access, secure in the knowledge that they are only using their authority to protect the public.
Unfortunately, as we have just seen with the IRS, they are quite capable of being political and biased at some level and having it spill over enough that they use their official capacity to cause issues. And despite my usual wariness towards expanding government power, I have to admit surprise that something this blatant has happened. I think direct comparisons to Watergate would probably be overblown, but when they say 'Nixonian', it rings at least a little true. All that there is really missing for that to be spot-on is the "amoral genius" in the central position.
The fact is, no matter how good your civil servants are *now*, they don't have to remain that way into the future. That's why limiting their power is always the right thing to do, even if it impairs efficiency a little. Authority, once given to a government, is rarely returned short of revolution, even if the need for that authority has evaporated. And it *will* be used, no matter what party is in office at the time, and no matter how much they campaigned on promising to not to do the same things.