I buy the "potential" issue. I have enough confidence in the leadership and the culture that I don't worry about it being abused in the near term, but eventually that could change. I actually do have a greater degree of trust in Google than I do other corporations or government agencies, though. I expect that's mostly because of the visibility I have as an employee.
The less they know about me, the better.
In the abstract I see that. But Google Now is useful... and I expect it to become vastly more useful. It's going to be interesting to see how this evolves over the next decade or so, whether most everyone decides that having an excellent personal digital assistant is worth allowing someone to know so much about them. At least it's shaping up that there will be competition... Now, Siri, Cortana, Echo...
And obviously Google is already using information it knows about users to make recruiting decisions so clearly they are using the data for more than just advertising.
Recruiting is advertising.
Suppose that I use an Android phone and I have all my web browsers signed in to a Google account. Google now has access to all my phone data, my contact data, calendar data, search history, and even info about websites that I go to directly w/o the help of google (thanks to Google ads)
Chrome can also tell Google everywhere you go even without the help of ads. It only does that if you turn on web history, though. Same with location. If you turn on location history, Google stores it. If not, Google doesn't get it. As for phone, contacts, calendar, photos, etc., that's true if you turn on backup for everything. If you turn off backup, the data doesn't go to Google. Of course, then you don't get the cross-platform always-updated calendar and contacts list, and if your phone gets run over by a bus it's all gone. Whether or not to use backup isn't a one-time decision, though; if you use it and then later decide not to you can use the privacy dashboard to delete stuff.
And Google does forget the data you ask it to delete. It's a good idea to check the dashboard periodically and wipe out anything you don't want to be there. You should probably do that if you haven't.