Crowd Wisdom Better At Predictions Than Top CIA Analysts 136
First time accepted submitter tkalfigo (1448133) writes "The Good Judgment Project is an experiment put together by three well-known psychologists and some people inside the intelligence community. What they aim to prove is that average, ordinary people in large groups and access just to Google search can predict far more accurately events of geopolitical importance than smart intelligence analysts with access to actual classified information. In fact there is a clearly identified top 1 percent of the 3000 predictors group, who have been identified as super-forecasters: people whose predictions are reportedly 30 percent better than intelligence officers."
Luck resets every time you guess. (Score:4, Interesting)
CIA uses Dirty Tricks Project (Score:2, Interesting)
CIA cannot believe a wisdom based output, they have to believe that their actions will change the outcome.
Reminds me of the Policy Analysis Market (Score:5, Interesting)
For example, I think a PAM system would have given us (and I mean everyone not just US policy makers) insight into how the events of the Arab Spring revolutions would evolve even if it couldn't have predicted the original flash point.
Re:Seems fishy (Score:4, Interesting)
Why do you think it is purely luck? When you have these wild discussion parties - things like "is a bright blob of pixels on a Mars Rover image a cosmic ray, a high-voltage dust-devil, light contamination of a camera box, a gas geyser", you will have an incredible combination of experts - everyone from geologists, ranchers, hill-hikers, photographers, astronomers. Geologists will tell you want can and can't come from the ground, ranchers and hill-hikers will tell you things they have seen and never seen, photographers will tell you what visual artifacts can appear on a camera, and astronomers tell you what can fall from the sky and can't, and what those falling things look like.
It's like solving a giant logic problem where everyone can cross off or tick what what they know. Eventually the set of possible answers reduces down to one or two.
Re:Reminds me of the Policy Analysis Market (Score:5, Interesting)
even if it couldn't have predicted the original flash point.
Funny you should say that, the diplomatic cable leaks showed [wikileaks.org] that high level western diplomats in Syria were concerned about a civil war erupting due to the severe "fertile crescent" drought fuelling internal migration from rural areas to the cities (10% of Syria's total population simply abandoned their farms due to lack of water). The drought caused food prices to rise sharply and food riots became a regular occurrence in cities across the middle east and North Africa.
"flash point" - Have a look at why that protester set fire to himself in the public square and why it resonated so strongly across the Arab world, it wasn't because they all logged on to FB and suddenly realised their governments were tyrannical. Predicting this sort of social unrest is like predicting an earthquake in LA, you can be pretty confident that your prediction will come to pass but have no idea when.
Re:Seems fishy (Score:4, Interesting)
Except this isn't how it works at all.
The wisdom of crowds works doesn't have anything to do with having experts. After all, the experts have no way of influencing the crowd. It is a well defined phenomenon that works when people's biases are pretty random, so mistakes cancel each other out. It's a lower quality estimation mechanism than a market, where people that are sure of their answer can be 'louder' than those that don't know said answer, and it lacks the feedback mechanisms of a market, but still, it is helpful to predict things based on widely available information. Ask the crowd information few of them have any idea about, and their result will suck.
So what does the average beating CIA personnel? That the CIA's biases are large enough to need quite a bit of quality control.
Now, having a 1% of the respondents be far better than the CIA experts probably means nothing. If I invite 3000 people over to guess how 10 coin flips will turn out, chances are one or two of them will guess all of them correctly, but that would not make them seers capable of seeing the future. how many people were worse than 30% worse than those same CIA experts?
Re:Well... (Score:4, Interesting)
The problem with Google's prediction algorithm is that it consistently overshoots. The story was on /. about a month ago [slashdot.org], as far as I can tell they're not only not predicting cases correctly, they aren't even attempting to distinguish between strains (how could they, they're predicting from search activity - flu victims rarely know their strain).