Like the federal government.
Well I think the idea that "If you have a hole in a system, it will be abused by malicious people" is a big part of the reason I'm uncomfortable with the federal government having access to people's personal info. Yes, there's the whole danger of dictatorship and secret police and bla bla bla. It's a real danger, but it feels far off. Far more immediate is the danger of... just some asshole that works for the NSA or FBI abusing the access. For all the assurances that "we have access to your data, but we promise only to look at it after we get a warrant from a secret court," you know that there's some dude at the NSA looking through email from people he went to high school with, just for kicks. And that's creepy and all, but if that guy is also a bit crazy and malicious, he can do some damage to people's lives.
So ultimately, the danger of the Federal government having access to your data is less that the Federal government is itself dangerous, but having access to private data without sufficient oversight is going to be abused by individuals within the Federal government.