Comment Re:I guess Elon Musk and Stephen Hawking (Score 1) 417
We are already seeing the huge risk in automating data gathering, almost precisely on the points you are mentioning too.
Loans are granted or denied based upon arbitrary criteria that is often full of biases that frequently even the loan officer rarely knows or understands. Try to ask one of these guys why your credit score is at a certain number, and they really can't honestly reply. They won't even be able to tell you how to improve your score other than some broad and vague generalities either... and no promise that if you engage in certain actions that it will have any impact at all. Furthermore, they can't even look at a particular credit score and history to tell you with any certainty that you will be approved... other than if your credit score (something very arbitrary) is very high that it is "very likely" or if it is low it is "very unlikely".
People get put on or removed from a travel "Watch List" with just as much arbitrariness... often because of messaging this kind of BIG DATA that you are talking about too. In that case it is more than just if you are going to get a fancy car, but basic liberties or where you may even be incarcerated simply because some computer algorithm thought your Slashdot post was a little too anti-government or scary based on keywords or something else you said.
There is massive misuse of these tools even now. Some of it has valid reasons for it being developed, but the blind trust of computers is already happening in a number of ways. This isn't a future concern but something right now that needs to be evaluated.... and like you said autonomy is not necessarily the issue at hand.