So much of the discussion about the potential harms of surveillance and data collection are unfocused, and often when you’re in one of those conversations you end up with nothing more than a vague notion that that someday, maybe, bad things will happen. But destructive algorithms — “weapons of math destruction” — already exist and are already harming us.
... I designate “weapon of math destruction” as algorithms with three primary characteristics — they’re widespread, mysterious, and destructive. Widespread because I only care about algorithms that affect a lot of people and have important consequences for those people. So if the algorithm decides whether someone gets a job, or goes to jail for longer, or gets a loan, or votes, then it’s a big deal. ... I call WMDs mysterious because the algorithms I worry about are secret. They come from hidden formulas owned by private companies and organizations and are guarded as valuable “secret sauce.” That means the people targeted by their scoring systems are unaware of how their scores are computed, and they’re often even unaware that they are being scored in the first place.