Follow Slashdot stories on Twitter

 



Forgot your password?
typodupeerror

Web Geniuses Or Web Dimwits? 164

Posted by Zonk
from the nothing-but-geniuses-here dept.
ScribeCity writes "The Washington Post has a provocative piece about online experiments at identifying experts. One wonders when someone will come up with a truly effective formula for measuring human intelligence — or take a stab at doing so — that exploits all the stuff people are publishing online." From the article: "This wisdom of the crowd could be outsmarted by what Michael Arrington, editor of the TechCrunch blog, recently dubbed the 'wisdom of the few.' Sites like PicksPal rely on input from the masses chiefly as a venue for auditioning prospective experts, on the theory that these virtuosos could provide even more accurate information and predictions than the crowd. 'If you figure out which ones did the best and get rid of the ones who have no idea, you'd do even better. Distill it down to the people who really know,' Arrington said."
This discussion has been archived. No new comments can be posted.

Web Geniuses Or Web Dimwits?

Comments Filter:
  • Or... (Score:5, Funny)

    by jo42 (227475) on Thursday October 19, 2006 @02:37PM (#16506525) Homepage
    Just get a chimp to throw darts at the wall...
    • Re:Or... (Score:5, Insightful)

      by UbuntuDupe (970646) on Thursday October 19, 2006 @02:51PM (#16506821) Journal
      If that chimp picked stocks that way, it would beat something like 80% of mutual fund managers (simply by virtue of not charging for his essentially random results).

      (I know, I know, "but not my mutual fund!")
      • by kalirion (728907)
        Scott Adams did an experiment on his blog in which he asked the readers for stock advice and came up with a small portfolio [typepad.com] based on the results. The portfolio did pretty well at first, before plummeting.
    • by R2.0 (532027)
      If they are posting stuff on the Internet and sound like an expert, they are most likely NOT an expert.
  • by garcia (6573) on Thursday October 19, 2006 @02:40PM (#16506555)
    In order to effectively determine the rate of experts vs. everyone else, you could simply scan through all previous Slashdot posts (while removing those prefaced by IANAL) and easily determine those that are experts.

    Make sure you are browsing at -1, *those* people are the real experts ;)
    • by RealProgrammer (723725) on Thursday October 19, 2006 @02:43PM (#16506641) Homepage Journal
      ... you sound like you could be an expert.
    • Re: (Score:3, Insightful)

      Make sure you are browsing at -1, *those* people are the real experts ;)

      You may joke, but these days anyone who questions the current pseudoscience-dogma-of-the-month tends to get modded -1 when they interject facts into the discussion, so you're not that far off.

      • Re: (Score:3, Interesting)

        You may joke, but these days anyone who questions the current pseudoscience-dogma-of-the-month tends to get modded -1 when they interject facts into the discussion, so you're not that far off.

        You know, that used to be more true. Then Digg came along and took away all the morons. Go check out some of their flamebait stories (politics would be a good start). If you don't echo the group view, you will be modded into oblivion. However, statements like "Bush is teh stupid!" will actually get modded up. Mo

    • by Cyclometh (629276)
      Maybe there's some way of harnessing /. moderation (and meta-moderation) data to get a candidate of likely experts.
    • Re: (Score:3, Interesting)

      by cultrhetor (961872)
      Places like /. are the basis for these sorts of developments - user-moderated, information-recommendation boards that rank opinion and content based on a number of criteria. Although a number of boards like this one fail or become shills (ePinions), those that survive are models for social recommendation researchers. Discourse analysis is a peculiar human trait, one that computers cannot (yet) accurately perform because our communicative practices are situated in unique, perspective-based contexts, so I'l
  • Never happen (Score:5, Insightful)

    by Quiet_Desperation (858215) on Thursday October 19, 2006 @02:41PM (#16506581)

    One wonders when someone will come up with a truly effective formula for measuring human intelligence

    It won't happen, not because it's not possible, but because some group or another will have a lower mean score, and the cries of racism, sexism, ageism, redbluestateism, culturalism, OSism, haircolorism, footsizeism, dicksizeism, or whateverism will drown out the truth.

    You know... the way it is right now.

    • I'm sure real scientists wish they could blame the failure of their theories on political correctness too.
    • by COMON$ (806135)
      Dont know, experts exchange does a pretty good job of weeding people out. Then again it is kind of like a MMoG in that you can rack up points pretty fast by just farming all day.
  • by Tackhead (54550) on Thursday October 19, 2006 @02:42PM (#16506613)
    > 'If you figure out which ones did the best and get rid of the ones who have no idea, you'd do even better. Distill it down to the people who really know,'

    ...and then disappear everyone who knows what they're doing, so you can hire clueless sycophants whose loyalty can be guaranteed.

    A sword cuts both ways, after all. I fear this tech.

    • by morgan_greywolf (835522) on Thursday October 19, 2006 @03:08PM (#16507187) Homepage Journal
      ...and then disappear everyone who knows what they're doing, so you can hire clueless sycophants whose loyalty can be guaranteed.


      Ooh! Just like U.S. Federal Government! Good idea!
    • by kthejoker (931838)
      There was a police academy in Connecticut (?) that administered an intelligent test to its prospective students. Obviously if you scored too low you wouldn't be admitted, but if you scored too high, you also were rejected - on the premise that smart people would get bored as cops, and either a) turn corrupt, or b) quit too quickly, thereby wasting the academy's time. Somebody sued them and actually lost the case, too.

      The moral: there are always other considerations.

  • Simple (Score:5, Insightful)

    by CrazyJim1 (809850) on Thursday October 19, 2006 @02:43PM (#16506623) Journal
    Make a group based moderation system, where you moderate in groups.

    Step1: Lets say Democrat/Republican. When a Rep mods something up, all other Reps see it modded up. If a Dem mods something down, no other Reps see it modded down.

    Step2: Identify posters who say stuff that gets modded up past a certain point. Lets say you get a point for the top 10 posts of each day. Then the posters with the most points are dubbed experts in their field.

    Its simple, and I'm suprised no one has done it before. It's like Digg in some ways, but vastly superior as groups don't bicker over what they declare as news, and it identifies experts.... maybe even political candidates.
  • :o\ (Score:3, Insightful)

    by TubeSteak (669689) on Thursday October 19, 2006 @02:43PM (#16506631) Journal
    I'm not an 'expert' in anything, yet I read far and wide enough to pick up lots of random & indepth tidbits that 'experts' have not heard about.

    Ever heard the joke about the phd professor who studied more and more about less and less, until he knew everything about nothing? Yea, many people would consider that professor an expert.
  • Not a Bad Idea (Score:4, Interesting)

    by AKAImBatman (238306) * <akaimbatman@gma i l . c om> on Thursday October 19, 2006 @02:43PM (#16506633) Homepage Journal
    There is a certain logic to this. How many times have "experts" told us screwy nonsense, and had lousy track records [dvorak.org], and yet the public at large retained them as experts? Sometimes, the untrained may be able to see things that the supposedly well-trained can't.

    Or to put it another way, it all becomes a set of probabilities. If person X has guessed the outcome of something (say, a football game) correctly 80% of the time, then you're safer betting on his predictions than you are betting on expert Y who is only correct 30% of the time. If you aggregate the probabilities and successes, you should be able to develop a model with a high probability of being correct. You'll never be able to gain 100% accuracy, but that's just the nature of the Universe [wikipedia.org]. ;)
    • by aafiske (243836)
      Wrong.

      Suppose we have everyone in the world guess the outcome of a 100 coin flips. Some fraction would be 100% correct. Repeat a few times. We've now winnowed down a pool of people who are excellent at guessing coin flips, right? It's safer to go with their guess than the guess of someone else, right?

      The fact is, _someone_ was going to be mostly right. However, there's nothing special about that person, they just happened to get lucky. Their previous luck does not affect their current predictive powers, whi
      • Suppose we have everyone in the world guess the outcome of a 100 coin flips. Some fraction would be 100% correct. Repeat a few times. We've now winnowed down a pool of people who are excellent at guessing coin flips, right? It's safer to go with their guess than the guess of someone else, right?

        Except that you're trying to predict a random event that can't be predicted. When a good baseball team defeats a bad baseball team, there's nothing random about it. When one candidate is elected over another, the ans

      • The problem with your argument is that it denies the very existence of expertise. You could apply the same argument to surgeons performing triple bypasses, and thereby "prove" that you're just as well off asking your mechanic to operate on you as you are going to Johns Hopkins.

        All you've really demonstrated is that it's possible to come up with false positives when determining expertise. This is not a surprising answer.
        • Not really. What he's demonstrating is that randomness (luck in this case) makes it more difficult to find the actual expert. In the case of surgeons, the difference in quality between the experts (999 out of 1000 operations are successful) and non-experts (0 out of 1000) is sufficiently big to reliably pick out experts even after a few observations. As there's no random component in doing a successful surgery (there is one in doing an unsuccessful one though), you only have to observe a successfull operati
    • Or to put it another way, it all becomes a set of probabilities. If person X has guessed the outcome of something (say, a football game) correctly 80% of the time, then you're safer betting on his predictions than you are betting on expert Y who is only correct 30% of the time.


      Hmmm...seems like the outcome of a football game could be predicted correctly at least 50% of the time if the predictions were chosen at random.
      • Re: (Score:2, Insightful)

        by Gospodin (547743)

        Sure, but the picks may not be random. If you accept that someone can be right 80% of the time, then they can be wrong 80% of the time and hence right only 20% of the time. Ironically, a sports gambler who is right only 15% of the time is more valuable than one who is right 75% of the time. You simply hear his picks then do the opposite. It's the one who picks 50% who is informationally worthless.

    • >> Could we call collective decision-making Democracy?

      Selective decision-making is good with Engineering -- I don't want to launch a rocket on what most people think.

      What is missing here is the "information dessemination." I think if a large group of people is made to understand the underlying facts, groups of people can make "wise" choices.

      But that is the problem; having the time, initiative and coordination to get people to understand the situation. Into this gulf steps the "expert." The expert dige
    • Re: (Score:3, Interesting)

      Not an original idea, but: what would be really fun is a website where people can go and bet on the likelihood of various future events. For example, everyone starts out with 1000 credits, which they can bet on events like, "The Republicans will retain control of the House in the November 7th election," or "Terrorists will detonate a nuclear warhead on American soil before January 1, 2015." Odds would vary over time as bets are placed, so that on average the winning side will make as much as the losing s
      • Such stockmarkets [uiowa.edu] already exist, and prove to be fairly accurate. Basic idea is to open a futures market for two events, say 'Rebs/Dems will obtain the majority in congress in November'. Set the liquidation value (profit) for the winning contract to 1 buck, and 0 for the losing side. Now let the trading begin: the market value for the contracts will determine the perceived odds for those events. Current situation [uiowa.edu]:

        Rebs more than 231 seats: last trade 4 cents

        Rebs between 217 and 231 seats: last trade 29 ce

    • "There is a certain logic to this. How many times have "experts" told us screwy nonsense, and had lousy track records [dvorak.org], and yet the public at large retained them as experts? Sometimes, the untrained may be able to see things that the supposedly well-trained can't."

      The problem is people have a carnal desire to feel superior to others, hence why people poke fun at others. And why kids get picked on in school and later in adult life in more subtle (and not so subtle) ways.

      The fact is people

  • by c0d3h4x0r (604141) on Thursday October 19, 2006 @02:44PM (#16506653) Homepage Journal
    'If you figure out which ones did the best and get rid of the ones who have no idea, you'd do even better. Distill it down to the people who really know,' Arrington said.

    I've always said that elections should qualify each voter's ballot to make sure the decision is made by the people who are best equipped to decide. The first page of a voting ballot should be a questionnaire that asks simple unbiased questions that require the voter to demonstrate knowledge of who or what they are voting on. "What does candidate X say their stance is on abortion?" "When did you first hear about initiative I-456?" "Please specify which political party each candidate below belongs to", etc. The score a voter gets on their questionnaire would then be used as a "weight" factor when counting their ballot, so that people who know the candidates and the issues better get more of a say, which is clearly how things ought to be.

    • Yes, but who decides which issues make the questionnaire?

      The questionnaire's authors would in-effect be defining the criteria for election.

      Maybe I vote for someone based upon whether or not they annoy the crap out of me.
      That's my prerogative.
    • The original eligiblity requirements for voting in the US were intended be the Founding Fathers to do just that. Unfortunately they where racist,sexist,and classist. I agree that it is odd that you need a test to be allowed to drive but not a test to vote. Maybe the best would be an unbiased video of a debate between all the people on the ballot that would be required viewing before going to the voting booths.
  • by davidwr (791652) on Thursday October 19, 2006 @02:45PM (#16506691) Homepage Journal
    1) Read old threads at +5, new threads at +2

    2) If a person has a lot of insightful/informative posts, check their posting history

    3) If they are consistently +3/4/5 informative/insightful, add them as a friend

    4) add points to friend's posts so they start out +2.

    OK, seriously, I don't do that but if I did, I'd see posts of "wise ones" and ignore posts from those that don't make the cut.
  • by Shimmer (3036) <brianberns@gmail.com> on Thursday October 19, 2006 @02:49PM (#16506771) Homepage Journal
    Let's say you have a pool of 10,000 prognosticators. You ask each one to pick the winner of 10 football games. The odds of getting all 10 correct are 1 in 2^10=1024. So out of the pool of 10,000 people, by random chance alone you're likely to get about 10,000/1024 = 10 people who pick all 10 games correctly. Are these people "geniuses"? No, they just got lucky during this particular trial. The odds of them getting game #11 correct are just 50-50.

    BTW, this can be used as the basis of a scam against the "geniuses" if you can convince them that they have special powers as a result of the trial.

    Moral of the story: Be very careful with statistics.
    • Re: (Score:3, Interesting)

      by richg74 (650636)
      This is actually extremely similar to the coin-flipping contest described in an appendix to one of my favourite books about the stock market (and other financial markets), Fred Schwed's Where are the Customers' Yachts?. Of the contestants who flip ten heads in a row, he writes, "they are the true experts, the ones who can't miss. They have their biographies written."

      Notice also the similarity to the fallacy underlying the articles one sees occasionally, along the lines of "Man Wins Lottery Twice Agains

    • Foiled by statistics again!

      You're as bad as the guy who takes issue with the statement, "We won't stop until all children are above average!"

      You're as bad as the guy who worked at MegaHuge Hedge Fund in the late 90's. His boss walked into the office one day all excited about a new way to measure risk, called "Downside Risk Quotient." He asked the guy how often the stocks in their portfolio were below their mean price, or what their "Downside Risk" was. The guy foolishly answered, "50% of the time."
  • "Democracy is based on the assumption that a million men are wiser than one man. How's that again? I missed something.
    Autocracy is based on the assumption that one man is wiser than a million men. Let's play that over again, too. Who decides?"

    And...

    "Always listen to experts. They'll tell you what can't be done, and why. Then do it."
    • And the best is when one wise man takes the input of a million and makes the best decision...
    • by Nephilium (684559)

      At least attribute...

      All from Lazarus Long, in the stories of Robert A. Heinlen...

      Nephilium

      The only religious opinion that I feel sure of is this: self-awareness is NOT just a bunch of amino acids bumping together! -- Jubal Harshaw in Stranger in a Strange Land

      • by spun (1352)
        Yes, the title of the post, "Obligatory Lazarus Long Quotes," was far too vague. Why, I bet most geeks have never even heard of Lazarus Long OR Robert Heinlein. ;-)

  • The common parlance term for the "expert witness" among lawyers is "the courtroom whore" -- Lots of fancy sounding credentials, gotta have the doctroal degree, and willing to say anything for a price. Totally worthless idiots in most cases.

    Besides:

    "If you ask enought experts, you can confirm any opinion or theory."

    Not sure who said it, but it's valid IMHO.

    • by Soko (17987)
      "If you ask enought experts, you can confirm any opinion or theory."

      Not sure who said it, but it's valid IMHO.


      So how many experts did we go through to get that one? ;-)

      Soko

  • Explains everything observed by these sites. I predict that a lack of science and math education will continue to result in people fruitlessly attempting to use past performances in predicting chance-based events (either because of true randomness or sufficient complexity to thwart casual analysis) to denote exports that will at some point start getting it wrong.
  • Nice scam (Score:4, Insightful)

    by DaveJay (133437) on Thursday October 19, 2006 @03:14PM (#16507361)
    Isn't this just that same old thing, where for each sporting event, you send a mailer to 50% of the people picking one team, and 50% picking the other, and whoever wins, that 50% of your original audience gets split between the two possible winners in the next mailing? Eventually you end up with a small audience, but they're CONVINCED you have a flawless sports betting "system" and pay you to learn it.

    Here, by pretending you're figuring out who the "experts" are, you're not diluting your audience with each round of guessing; instead, you're diluting your potential pool of "experts" (or systems), and eventually everyone decides that person X is always right, when really odds were that at least one person in a large pool of guessers would guess right 100% of the time.

    Past performance is no guarantee of future performance, people.

  • by Col. Klink (retired) (11632) on Thursday October 19, 2006 @03:18PM (#16507431)
    Weird. The phrase Wisdom of the crowds [worldcat.org] was coined by James Surowiecki as the title of his book (see also wikipedia [wikipedia.org]). The premise was that crowds, on average, can do better than a committee of experts. It's not that there is someone always in the middle, it's actually the highs and the lows aggregated that make sense in the wisdom of the crowd.

    This sounds like the old scam. Pick 1000 people. On day 1, send 500 of them a prediction that stock A will go up and send the other half a prediction that the stock will go down.

    On day 2, the stock either went up or down. Either way, you made a correct prediction to 500 people. Split the 500 and send two more predictions on an all new stock.

    Keep repeating this. On the fifth day, you'll have 75 people who have seen you make 5 perfect predictions in a row. Now ask each of them for $10,000 to invest in your next prediction...

    Just because one person happens to have hit the mean each time doesn't mean he's got "the knack". Statistically, there's sure to be someone whose guesses approach the mean. But that doesn't mean that their next prediction is any more likely to be accurate.

    Stick with the aggregated mass knowledge.

    • Re: (Score:2, Informative)

      by Lurker2288 (995635)
      Yeah, I sort of thought the whole idea behind 'wisdom of crowds' is the fact that you aggregate enough data to cancel out the individual biases and result in a relatively accurate conclusion. If you pull out all the "experts" who make the correct call in one trial, don't you lose the correcting power of the group? How big and intellectually diverse does a 'crowd' have to be?
    • by Sean0michael (923458) on Thursday October 19, 2006 @04:02PM (#16508243)
      Exactly. Having a pool of experts does not mean you will have a better outcome. The odds of any one expert being correct every time are very slim. But the odds of a group being correct every time are actually better. For those that watch the football announcers (our experts) make predictions about who will win today, they disagree and none have perfect records. But when it comes to predicting spreads, the guys controlling how many points one team will win by are much more accurate because a large number of people, all with private knowledge and information plus sharing some general knowledge, all weigh in on the outcome. This diverse group (which includes experts) generally gets the correct spread (if they don't, the sports gambler in charge is losing lots of money).

      Experts are great, and their knowledge is valuable. But in making certain kinds of decisions, it is better to tap into the Wisdom of Crowds.

    • I've had my doubts about the idea that if you take the average of all the guesses a crowd makes it comes out to be near correct. So last time I was at an event that had a "guess the number of jellybeans and win a prize" contest I asked the organizer for all the stubs at the end of the evening. I went home and punched all the guesses into a spreadsheet.

      The mean was in the ballpark but not accurate enough to win the prize. When I took the median I got an answer that was as accurate as the best guess. So

  • by not already in use (972294) on Thursday October 19, 2006 @03:27PM (#16507587)
    One thing you have to remember: Perception > Reality. Speaking intellegently and writing intellegently is usually enough to convince someone that you actually know what you're talking about, if you're audience is ignorant or naive. That makes for a lot of percieved experts in the field of technology. Take the example of an internet born initiative to ban dihydrogen oxide in some county California http://www.dhmo.org/facts.html [dhmo.org]. Read this. If you haven't already heard of this, well, dihydrogen oxide is water. See how easy it is to convince a bunch of soccer moms they need to ban water? (Or that apple needs to abandon hardware... hehe)
  • And now, everybody feels pressured to post. Great job- post an article about finding internet experts on slashdot.
  • by Cherita Chen (936355) on Thursday October 19, 2006 @03:47PM (#16508001) Homepage
    Didn't Slashdot solved this problem with their moderation scheme? Oh, wait, nevermind... that would mean "Karma Whores" would qualify as experts. Nevermind.
  • The Emperor (Score:2, Interesting)

    by entropy123 (660150)
    This quote, headlined today on google, is instructive: It has always been the prerogative of children and half-wits to point out that the emperor has no clothes. But the half-wit remains a half-wit, and the emperor remains an emperor. - Neil Gaiman I find that 'Experts' are largely chosen based on qualities other than their expertise. Usually they have good personalities and make friends easily, especially with leadership. The leadership prefers to pick and talk with experts who generally agree with the
  • Okay, so what are the odds someone can flip a coin ten times in a row, and have it turn up heads every time? 1 in 1024. So now say we have 10,000 people try this and capture each on video, and we find which actually succeeded in turning up heads 10 times in a row. Now, what are the odds that person will flip another heads on their 11th try? It's still only 1:1 of course. However there are those that will believe, after watching the video, that the odds are better than that - the whole "gut feeling" thi
  • problems can be divided into three categorys
    1) easily knowable answers, like who is the 10th president, or angelina jolies boyfriend

    2) hard to know answers, like how may diff types of beetles live in california - with enough money, you could answer this, but it would take alot of work

    3) unknowable answers, like what interest rates will be this year, or what the stock market will do tomorrow (obviously unaswerable, cause if you could, you would make gates look like a pauper), or what will happen in Iraq.

    The
  • Somebody tell Jimbo Wales!
  • > Web Geniuses and Web Dimwits

    Don't you mean Webiniuses and Webimwits?
  • by quisph (746257)
    If 10,000 people each flipped a coin ten times in a row, chances are good that nine or ten of them would have it land heads-up each time. Have identified the "expert" coin-tossers, then? I don't think so.

If imprinted foil seal under cap is broken or missing when purchased, do not use.

Working...