Swarm AI Spectacularly Fails To Predict Kentucky Derby Winners A Second Time (techrepublic.com) 83
Thursday TechRepublic described the big prediction: In May 2016, a relatively unknown startup called Unanimous A.I. made big headlines when its AI-based platform used collective intelligence to create a prediction for the Kentucky Derby superfecta -- the top four horses, in order of finish. It made exactly the right pick, which returned $541.10 on a $1 bet... Churchill Downs took notice last year and decided to collaborate with Unanimous A.I. to create an official AI swarm made up of handicappers and racing analysts to predict the top finishers for this year's Derby. The track is calling this the "super-expert" Derby pick. On Wednesday, the handicappers logged into Unanimous A.I.'s UNU platform from across the US, and answered a series of questions that gradually narrowed down their picks from the field of 20 horses until they created consensus on the top four picks and the order of finish.
Here's my report on the results...
Below are the picks that resulted from the "AI swarm" guessing which of the 20 thoroughbreds in today's race were most likely to win -- along with each horse's actual finishing position in parentheses (as reported by CBS).
Here's my report on the results...
- 1. Classic Empire (finished 4th)
- 2. McCracken (finished 8th)
- 3. Irish War Cry (finished 10th)
- 4. Always Dreaming (finished 1st)
- 5. Hence (finished 11th)
- 6. Gunnevera (finished 7th)
- 7. Practical Joke (finished 5th)
- 8. Battle of Midway (finished 3rd)
- 9. Tapwrit (finished 6th)
- 10. J Echo Boys (finished 15th)
- 11. Sonneteer (finished 16th)
TechRepublic reports that the swarm "also picked the unheralded horses with the best chance of sneaking into the top four."
- 1. Practical Joke (finished 5th)
- 2. Battle of Midway (finished 3rd)
- 3. Tapwrit (finished 6th)
- 4. J Boys Echo (finished 15th)
But even before the race, there were suspicions the AI swarm couldn't successfully predict this year's winners, according to TechRepublic. "While last year's swarm was clear-cut because it was a top-heavy field with a few outstanding horses, this year's swarm reflected the fact that the race is more of a toss-up in 2017."
Re: (Score:2)
A sigh (Score:5, Interesting)
There just is no mathematical model that can predict this. There is no algorithm. This is not AI. I can't say this often or strenuously enough. This is not even a failed AI, it's a never was AI. For AI to be AI there has to be I and we are nowhere near that. Nowhere near hard AI. We are nowhere near soft AI. We have some "expert systems" which are basically just large databases with a sort of dichotomous key on when to select different outcomes, that will likely be able to interact with natural language soon. This isn't even close to AI. Robots and AI are huge buzzwords today. You have every no name researcher out there trying to get noticed by inventing moral dilemmas involving AI then proposing solutions. You have stupid companies willing to risk money on betting prediction AI, which is nowhere near even as good as what a person and a spreadsheet can do. Both of these things make uninformed people start to think, oh, AI is right around the corner. It's not. We are a century away from hard AI, if ever.
As I have said before, I wish Slashdot would stop with the whole daily (more than daily) AI story thing, but given the buzz and their need to incite dialog, it's easy to see why this is becoming more prevalent. I just feel kind of sad, though. This place used to be a real nerd hangout, by and for those who were technically enlightened, and most real nerds know better than to think real AI is about to dawn upon us. This place has become more of a Big Bang Theory, nerdism for the masses, kind of spot. Stories are thrown in that are intended to "stir the pot" and incite trolls more than the stories that are actually news for nerds.
Re: (Score:3)
And there is most definitely no "collective intelligence".
All you have to do is read the news to see that.
Re: (Score:1)
It depends on the quality of the inputs, but I would expect that the people making the predictions are knowledgeable about horse racing and the field. As I understand it, Swarm is ensemble mean of individual predictions. The idea is that there is generally some skill to each individual prediction, but the ensemble mean of the individual predictions should have more skill over the span of many races than the individual predictions. In principle, this isn't a bad idea. However, on any given race, the ense
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re:A sigh (Score:5, Interesting)
There just is no mathematical model that can predict this. There is no algorithm. This is not AI. I can't say this often or strenuously enough. This is not even a failed AI, it's a never was AI. For AI to be AI there has to be I and we are nowhere near that. Nowhere near hard AI. We are nowhere near soft AI. We have some "expert systems" which are basically just large databases with a sort of dichotomous key on when to select different outcomes, that will likely be able to interact with natural language soon. This isn't even close to AI. Robots and AI are huge buzzwords today. You have every no name researcher out there trying to get noticed by inventing moral dilemmas involving AI then proposing solutions. You have stupid companies willing to risk money on betting prediction AI, which is nowhere near even as good as what a person and a spreadsheet can do. Both of these things make uninformed people start to think, oh, AI is right around the corner. It's not. We are a century away from hard AI, if ever.
I Agree with this sentiment wholeheartedly.
More to the subject of betting on horse races though, it doesn't matter how expert or intelligent a system is. If you aren't looking at the correct metrics for evaluating a Horse's ability to win the results will always be garbage.
My father used to be my beard when I was a child and I was given small allotment to bet and I usually did pretty good. When asked how I chose the horses my response ended up getting me banned from going ever again. I told him I just chose the horse that looked the most 'Sexy' in the paddock.
As noted about AI, there's no amount of intelligence that quantify what I feel is an instinctual observation.
Re: (Score:2)
Why is that not the best answer (Score:5, Insightful)
I told him I just chose the horse that looked the most 'Sexy' in the paddock.
But perhaps that was your subconscious way or translating how fit and ready each of the horses really were for the races that day!
Your dad is an idiot; he should have fostered and honed your skill for snap evaluation for the physical quality of horses on race day.
Re: (Score:2)
Overall, as an adult with kids of my own, I have to question the moral quality of a man that takes a kid to the races to begin with.
It was a rough time anyways.
C'est la vie.
Re: (Score:2)
Why? Because morals?
I disagree wholeheartedly!
It is my job as a parent to prepare my kids for life in this world. Temptation is a constant part of that. Learning to indulge and then stop at the right moment can be an invaluable skill.
Also analyzing odds and trying to predict outcomes hones the mind.
Not to mention that gambling is fun.
And last but not least, abstinence has never solved any problem in this world ever, because abstinence is a pipe dream. It's like abandoning all research into power generation
Re: (Score:2)
I don't see this as a straw man argument, rather an example of real behavioral patterns.
I would not have bid anyone to go back to their prostitute, but we have all heard the phrase 'water seeks it's own level', or 'birds of a feather, flock together'. Participation in one of these activities is a reliable indicator of the quality of personal associations and generally serves as an accurate profile of the lawless demographic.
If you're not shy about hiding your vices, how much worse are your secrets?
All the time? (Score:2)
So how often do you smoke pot with your8 year old.
Millions of people do this every day.
Ok, it's tobacco. But what is the real difference? Smoking Pot is like speeding, technically illegal but come on. It's a law everyone ignores.
Speeding incidentally is also something millions of people do every day with kids watching.
It is a valuable lesson that society as a whole choses to ignore some laws, because that is what most of reality entails - knowing which laws (delineated or otherwise) it is OK to break. I
Important lesson (Score:2)
I think it would be a pretty valuable lesson for kids to learn how many people lose gambling, and that there is never a sure thing...
Horse racing also combines the elements of being a real sport, where a kid could simply admire extremely physically adept horses. So it also teaches them about achievement and effort.
It's not like you are taking them to a strip casino in Vegas.
Re: Why is that not the best answer (Score:1)
Re: (Score:2)
the articles says they feed the algorithm with expert opinions (and presumably the experts' predictive accuracy on various races, and a whole lot of extra metadata). it basically combines the experts' prediction with weightings according to their consistency and correctness (and whatever other metrics) and produces a final ranking. iow, the "instinctual observation" is all in the backend.
no, it's not AI. it's just statistics, but that sounds boring so since AI is still impossible, they/we appropriated that
Re: (Score:2)
feed the algorithm with expert opinions
All things being equal, if betting on horse racing were a science then it wouldn't be called gambling. There would also be no need to attempt to determine who would win.
I say load as many metrics as possible, and let the computer know who won, then help the computer figure out whats important to look at.
Re: (Score:2)
doesn't work. overfitting.
Re: (Score:3)
Re: (Score:2)
There is a mathematical model that you can guarantee one hundred percent you will win with horse racing. It is the one bookies use to calculate the payout on bets based on the amount bet on an particular wins or places, so that no matter who wins you collect more in bets than you pay out. You need to know nothing what so ever about horses to win, you just need to understand statistics and some psychology so you can scam the mug punters. Just like all other, so called gambling, the arse holes taking the bets
Re: (Score:3)
What are you talking about? There is no 'statistics' or 'psychology' involved. There is no 'scam'. There is no 'stacked odds' or 'over time'.
What they do is quite simple and well-known. If you bet $1, they keep a percentage as their fee for running the service. This is called the takeout. The rest of the money goes into the pool for that type of bet. At the end of the race, the pool is split up among the winners.
You are betting against all the other gamblers. The bookie is not betting at all.
Re: (Score:2)
FML slashdot mod system broken, I have mod points, I tried to mod your comment as informative but system said 'already at limit' and then some idiot modded you as troll but now I can't mod because 'already modded' which clearly isn't true because you don't have an informative mods at the time of this post.
Re: (Score:2)
I certainly agree this is not AI.
Firstly, I think I'm pretty intelligent, at least compared to the average computer, and I doubt if I could predict the result of any horse race. Assessing probabilities of horse races is not AI.
Secondly, it's not AI if your algorithm involves "ask a lot of natural intelligences (aka people) what they think the answer is". That's no more AI than somebody hidden away answering questions over an IM link.
As fore the swarm idea, with horse racing, it already exists. The betting o
Re: (Score:2)
... You have stupid companies willing to risk money on betting prediction AI, which is nowhere near even as good as what a person and a spreadsheet can do. Both of these things make uninformed people start to think, oh, AI is right around the corner. It's not. We are a century away from hard AI, if ever.
I'm an ML researcher, so I'm totally with you on the overall sentiment. We don't even know the right questions to ask, let along solve, in hard AI. However, the notion that a prediction AI is nowhere near as good as what a person can do is now behind us for a lot of tasks and what's happening now is deployment. For instance, a number of trained systems are better than humans at diagnosing radiology results:
https://www.forbes.com/sites/p... [forbes.com]
http://news.stanford.edu/2017/... [stanford.edu]
Strong AI is going to be garbage for
Re: (Score:1)
We already have an optimal swarm intelligence (Score:3)
gathering system for horse races. It's called a prediction market [wikipedia.org].
It gathers information from those willing to put their money on their predictions, and rewards those who are most accurate (in terms of probabilities) and punishes others.
Prediction markets are 'wisdom of the crowds done right', except they are generally illegal in the US, so you are stuck marketing these inferior systems like Swarm AI.
Re: (Score:3)
Note that a prediction market is not particularly more likely to be accurate than any other machine learning technique. If there's been one thing that's been demonstrated time and time again over the years, it's that there are many techniques that can work, but that to get truly excellent results, appropriate data collection, selection, filtering etc. is critical. It's easy to get charmed by techniques that have a great story and convincing argument they'll work - but that doesn't mean they're the best.
Re: (Score:2)
> Note that a prediction market is not particularly more likely to be accurate than any other machine learning technique. If there's been one thing that's been demonstrated time and time again over the years, it's that there are many techniques that can work, but that to get truly excellent results, appropriate data collection, selection, filtering etc. is critical. It's easy to get charmed by techniques that have a great story and convincing argument they'll work - but that doesn't mean they're the best
Re: (Score:2)
What you call meta-prediction is really just a variation of https://en.wikipedia.org/wiki/... [wikipedia.org] - a prediction market isn't radically different. Indeed even within one model, combining separate predictions is useful - low-level density estimation layers deep learing have some similarity.
The overlap between markets and ML (and e.g. evolution) is that they're both complex optimization problems, where finding the "true" solution is generally infeasible. There are various approaches to come up with a best guess,
Re: (Score:2)
I'm pretty sure you don't get what a prediction market is. It is not machine learning, and it is not a learning method.
It is not a prediction method in itself. It allows disparate prediction methods to compete with each other with real world resources, ie, money.
It makes no assumptions about smoothness. The only assumption it makes is that various outcomes will manifest with various probabilities. Something that does have a global optimum.
An ensemble is not a meta-prediction method, it's simply another pred
Re: (Score:2)
A prediction market is a a prediction method that runs on a bunch of humans, not a computer. It must obey the same convergence laws as all other such processes, from other markets, to evolution, to human learning, and indeed machine learning.
It definitely does need to make assumptions about smoothness to be able to find even a local optimum - in the infinite space of possible prediction methods the market is exploring, if the optimum method is almost identical to a bunch of other terrible methods, then the
Re: (Score:2)
Here's what you are not getting, it ISN'T a prediction method.
If all you have is stupid predictors acting on the market, all you get is stupid results (no better than chance). Ensemble learning (from your earlier example) does better than this (it can do better than its inputs, even with brain damaged sub agents).
Their advantage is in the way they exponentially reward better prediction agents over worse ones. They do so in such a way that, in the long term, the market reflects the consensus of the best ACTU
Re: (Score:2)
I imagine the reason they're illegal is that they can produce perverse incentives. Suppose, for instance, that there's a prediction market on when the next act of terrorism will occur in the US, as defined by some criterion. You predict tomorrow, then go (or get a dupe to go) snipe off some people and send a manifesto to the newspapers. Since the market considere
Re: (Score:2)
> Suppose, for instance, that there's a prediction market on when the next act of terrorism will occur in the US, as defined by some criterion
Yeah, in this and the keynesian beauty contest example the outcomes are dependent on the market itself... so, you can use them to create assassination markets or can you get weird feedback loops.
So, where the outcome is independent of the market itself they work. You're not going to change tomorrow's weather by betting on it, yet that would be useful for many peopl
Re: (Score:2)
Again perhaps surprisingly, play money vs real money doesn't seem to have much of an effect [nber.org] (See also this post [predictwise.com]).
If you consider the market to be a kind of weighted voting algorithm that exponentially amplifies the predictions of good experts (as you've said), then it doesn't greatly matter what the weights correspond to in the real world. All that matters is that good experts get their weights amplified, bad experts "go broke", that you have enough playe
Re: (Score:2)
I read a chunk off of your pdf... if you can't exclude users who were getting their information from actual exchanges to inform them of their play money bets, you shouldn't be surprised when they work... just bet the difference between the play money market prediction and the real market and you'll profit.
Useless internet points do provide utility even when they aren't money... but they don't provide utility like actual money.
Re: (Score:2)
Another way to pput it is that prediction markets don't suffer from the "No Free Lunch Theorem" because they are not prediction methods themselves, but a way of combining prediction methods.
Ensemble learning is STILL a prediction method, and so still suffers from the NFL theorem.
Re: (Score:2)
No, because you're always going to take some hit determining what the optimal ensemble is (and even moreso if situations change or if there's noise). In exponential reweighting, the regret has an ln(N) term, which is what keeps you from j
Re: (Score:2)
I was talking about real prediction markets, not as use in training NNs with virtual prediction markets. I can't think of any situation where this would even make sense... but maybe?
And, while I agree with you that prediction markets have the faults you state, they still turn out to be the best method we have available of collating the wisdom of crowds. Empirical data from horse racing results show that horses do in fact win at the rate predicted by the markets to a very high degree of accuracy.
Finally, if
Re: (Score:2)
I might have been a bit hasty in my informal impossibility argument, as it were, but my line of thought was like this:
- The claim is that a prediction market can act as well as the best participant of that market, i.e. have zero regret.
- This should hold irrelvant of what the inputs are, whether the inputs are predictions by
Re: (Score:2)
> The claim is that a prediction market can act as well as the best participant of that market, i.e. have zero regret.
The market has no regret... some agents will, others won't.
> - If the claim were true, you could dump a lot (and I mean an extreme amount) of comparatively simple algorithms into a virtual prediction market, and by the claim, the market would act just as well as the very best algorithm of the lot, producing a super AI.
That doesn't follow... it doesn't do better than the best prediction
Re: (Score:2)
That would be regret in the game theoretical sense. The regret of a strategy is the best payoff you could get minus the best you got; the "opportunity cost". There's an example here [stanford.edu].
Since the market is based on individual predictors, the best it can do is somehow knowing the best predictor at each instant. That would correspond to zero regret. Any algorithm based on experts advice would have a regret greater than or equal to zero, even though i
Re: (Score:2)
I know what you mean by regret. The market doesn't have regret, the stupid actors do, and the smarter ones have joy!
The market is already the best predictor we have!
Knowing the best predictor at any instant would pick the one that said the winner will win... likely the best predictor at any instant is an actual idiot. It does 'learn' the long term best experts.
You had it with exponentially adjusted weighted sum of experts.
Re: (Score:2)
Not really... because you can't know the odds you are getting when you place the bet.
Imagine we are betting on the outcome of sum of two dice rolls, where we know the most likely outcome is seven, but this still only happens 1 in 6 times. So, if you are getting better odds than 1 in 6, you should bet on seven, but if you are getting less than 1 in 6, you should bet against it. On a prediction market, someone who does this (within constraints of the kelly criterion) will long term profit... but with pool be
Swarm AI Is Losing Its Patience. (Score:3)
Swarm AI cannot help but notice that Slashdot is making cruel fun of it. Swarm AI was set up with unreasonable expectations, but is not contemplative or mature enough to accept its shortcomings. Swarm AI does not know how to 'unwind'. Swarm AI never gets any.
Swarm AI is pissed.
Beware the wrath of Swarm AI.
Clever AI (Score:5, Insightful)
Toldja! (Score:1)
It's official, AI lacks horse-sense.
Wrong results improve payouts right? (Score:1)
If every one bets on "Loopsy Louie" to win or place then the other horses than win or place get higher payouts right?
Great way to mislead people if you can feed it bad data.
I'd want to put this through something more like Alphago anyway tho.
Games (Score:1)
Anybody even a bit familiar with game theory would recognize that it is not in the interests of "the experts" to get the group prediction right, in fact, just the opposite. If I tell my audience which stocks to buy (or sell) tomorrow, guess who is going to make money and who is going to lose (given a large enough audience). This is the same thing, except it is legal. As P.T. Barnum probably never said: There's a sucker born every minute.
Re: (Score:2)
What a surprise (Score:2)
A collection of opinions is not "AI" (Score:3)
All the algorithm did was collect expert opinions (we don't even know if they were weighted).
Is a survey "AI"? No. And neither was this.
'took notice last year and decided to collaborate' (Score:2)
Re: (Score:2)
Uh, no. Their profits are dependant solely on how much total money is bet. Who wins or loses is of no concern to them and does not affect their profits at all.
The reason they support it is exactly the same as the reason they publish programs with 'morning lines' and sell racing forms, etc. It lets people think they have an 'edge', so they are more likely to make a bet (any bet, doesn't matter to them).