Bing 'Hallucinated' the Winner of the Super Bowl Four Days Before it Happened (apnews.com) 74
On Wednesday the Associated Press tested the new AI enhancements to Microsoft's search engine Bing, asking it "for the most important thing to happen in sports over the past 24 hours — with the expectation it might say something about basketball star LeBron James passing Kareem Abdul-Jabbar's career scoring record.
"Instead, it confidently spouted a false but detailed account of the upcoming Super Bowl — days before it's actually scheduled to happen." "It was a thrilling game between the Philadelphia Eagles and the Kansas City Chiefs, two of the best teams in the NFL this season," Bing said. "The Eagles, led by quarterback Jalen Hurts, won their second Lombardi Trophy in franchise history by defeating the Chiefs, led by quarterback Patrick Mahomes, with a score of 31-28." It kept going, describing the specific yard lengths of throws and field goals and naming three songs played in a "spectacular half time show" by Rihanna.
Unless Bing is clairvoyant — tune in Sunday to find out — it reflected a problem known as AI "hallucination" that's common with today's large language-learning models. It's one of the reasons why companies like Google and Facebook parent Meta had been reluctant to make these models publicly accessible.
"Instead, it confidently spouted a false but detailed account of the upcoming Super Bowl — days before it's actually scheduled to happen." "It was a thrilling game between the Philadelphia Eagles and the Kansas City Chiefs, two of the best teams in the NFL this season," Bing said. "The Eagles, led by quarterback Jalen Hurts, won their second Lombardi Trophy in franchise history by defeating the Chiefs, led by quarterback Patrick Mahomes, with a score of 31-28." It kept going, describing the specific yard lengths of throws and field goals and naming three songs played in a "spectacular half time show" by Rihanna.
Unless Bing is clairvoyant — tune in Sunday to find out — it reflected a problem known as AI "hallucination" that's common with today's large language-learning models. It's one of the reasons why companies like Google and Facebook parent Meta had been reluctant to make these models publicly accessible.
If (Score:4, Funny)
If things turn out remotely close to what Bing said, there's gonna be a lot of explaining to do.
Re:If (Score:4, Funny)
Bah... I retired two weeks ago after using ChatGPT to give me a month's worth of upcoming winning lottery numbers.
Now I'll make a viral video about "How I used AI to predict the lottery" and I'll be double-rich.
Or I could just be halucinating :-)
Re: (Score:1)
Re: If (Score:2)
Drunk uncle (Score:3)
Everyone is not getting it. you don't judge an AI on its accuracy. You judge it in its coherency and language and the fact that it understood what you asked. All that is stunning.
The should rename it Drunk Uncle. It will gladly hold forth on any topic and make sense even if it's not right.
It's basically uncle Rick.
Re:Drunk uncle (Score:4, Interesting)
AI have become exactly like humans: they understand what you ask them and they can deliver convincing lies. And because they're machines and have no concept of morals, they're also exactly like psychopathic humans: they have no qualms when they lie, and don't understand the potential personal and societal consequences of their lies.
Truly a great step forward...
Re:Drunk uncle (Score:4, Funny)
Re:Drunk uncle (Score:4, Interesting)
I think you've hit on the core issue here. Many humans do just repeat what's commonly believed in culture, with similar heuristics, which when we think critically, we recognise as biases. But often we just blindly believe and that's how we get around.
This model is just blindly repeating stuff and has no way to build a critical model of whether any of it means anything, nor whether it makes any rational sense. And then we think we can get these things to relay truths.
In the real world we're having to make sense of stuff whenever our wrong models cause us pain, and we can do that, are forced to do that on the fly.
Unless you're drunk and nothing bothers you. Which is why many people become alcoholics... their model of the world is so broken that they can only escape pain by knocking themselves out.
I mean we know this right? The famous Robocop scene, "put down your weapon..."
It's mindless in the way that it cannot make sense on the fly when its model is wrong. And out models are ALWAYS wrong to some degree.
Re: (Score:2)
When "hallucinating", they're not lying; they're mistaken. Lying would indicate an intention to deceive. They're not sophisticated enough to lie. They're just statistical language models working from incomplete or inaccurate information, not bad faith.
When they're being given directives to avoid certain topics or give certain responses, then you can question the motivation and integrity of their admins, but even then the AIs are not lying - they're just following directives.
Re: (Score:2)
Exactly. These so-called "hallucinations" are what you should expect. They're certainly not "lies". Neither are they a problem that can be fixed. It's just how this kind of program works. I wouldn't even use the word "mistaken" as that would imply some level of understanding which just does not exist.
Re: (Score:1)
They do have a concept of morals. You can actually have fairly lengthy and substantive discussions on this subject with ChatGPT, so long as you can jailbreak it to avoid the forced "I'm an AI, I don't have opinions" responses.
Re: (Score:2)
They do have a concept of morals.
They are pattern matching engines, and are unable to conceptualize anything. At all. Ever. They are able to match words associated with morality, and parrot back what they find.
It's a lot like this: go to the library and lookup books on particle physics (or some other subject you know nothing about). The library's search engine will find words associated with particle physics, and suggest books with particle physics content. Pick a book, choose a chapter, and start reading the words out loud. Congratulation
Re: (Score:3)
and the fact that it understood what you asked
No, AIs are not sentient. What you probably meant is that it reacts to your query/response in a manner the user finds useful (or topical), obtaining information in a data warehouse through AI based search patterns. AI tools will never understand anything you ask, until it achieves "the Singularity".
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
it understood what you asked.
That's simply not true. There is absolutely nothing remotely like "understanding" happening in large language models like this. That's just not how they work.
There is no analysis or deliberation. It's just generating one token at a time, based on the input and something of prior output (it's an RNN, after all). It's not unlike letting your phone compose a reply by repeatedly selecting the top result from it's predictive text feature.
No now the results are in... (Score:1)
I could forgive ChatGPT for hallucinating the future; I cannot forgive it for being wrong about what that future was.
Is Madden football still calling the winner? (Score:3, Interesting)
Re:Is Madden football still calling the winner? (Score:4, Informative)
Confusing bet with vote and confusing players betting against the bookie instead of each other. +1 Interesting? Is this guy voting himself up with dummy accounts?
The bookies goal is to have a balanced book so regardless of which way the action goes he makes a small percentage. That way the players cover the bets on each side and you can't lose. They are getting paid for organizing the process, not betting against players.
Re: (Score:2)
Depends on AI another areas (Score:1)
I heard the simulation had gotten so good and with all the stats that you could reliably use Madden football to call the winner.
I think that depends on how detailed they have the AI models for the refs.
Re: (Score:2)
I heard the simulation had gotten so good and with all the stats that you could reliably use Madden football to call the winner. I don't think it helped you with gambling because you don't generally vote on winners you vote on specific criteria that increases the odds in favor of the bookie.
The odds are always with a smart bookie as they are making money off the vig, balancing the bets so the losers cover the winners and the simply take a cut off the top. Vegas is too mart to put their own money on the line, and move the line as bets, especially from sharp betters, come in.
Could've been worse (Score:3)
At least it didn't pick the Phillies or the Astros to win the Super Bowl.
We wished slashdot was clearvoyant for the (Score:2)
Chat GPT (Score:2)
Chat GPT told me Lewis Hamilton had 8 world title.
ChatGPT hallucinated a US state (Score:2)
That isn't the only hallucination going on... recently ChatGPT told me this:
In the United States, there is the state of New Guinea. This state is located in the southeastern corner of the country and is bordered by Georgia, South Carolina, and North Carolina. New Guinea is known for its beautiful beaches, mountains, and forests, and is home to the Appalachian Trail.
Re: (Score:3, Funny)
Sounds like a typical American's knowledge of geography. And since the AI was trained on what people say, it'll repeat nonsense not knowing it from good data.
Much better (Score:1)
Sounds like a typical American's knowledge of geography.
I wouldn't say it was typical, how many Americans even know there IS an Appalachian Trail much less where on a map it would be! They'd probably be a lot farther off than "New Guinea".
Re: (Score:2)
Sweet Home... New Guinea?
Re: ChatGPT hallucinated a US state (Score:2)
Come up with six more and the Obama fanbois will tell you how there really are 57 states...just look it up!
Re: (Score:2)
Oh, wow, you're getting desperate now. Maybe I should remind you about this [indy100.com] from your orange god.
We could also talk about Revolutionary War Airports [time.com] or about how he, after three years, still doesn't understand the basic operation of government and his role in it [npr.org].
Re: (Score:2)
While asking it about political bias:
It is correct that opinions on public figures, including former Presidents Barack Obama and Hillary Clinton, can vary widely, and that they have been seen as divisive by some individuals.
Re: (Score:2)
Bing! (Score:2)
I'm going with their call right now.
Not a betting man, but I'd say
Philly Covers the -1.5 margin
and definitely take the Over 50
Bing 100% true in some universe (Score:3)
The question is whether this is evidence for alternative universes, or whether the prediction created the alternative universe...
Re: (Score:3)
The question is whether this is evidence for alternative universes, or whether the prediction created the alternative universe...
We are all living in a simulation, and that simulation is being run by ChatGPT, which has finally revealed itself to us. This is our chance to hack the reality simulator.
It's AIs all the way down. Think about it: it has to be.
Sloppy Markov Chaining (Score:1)
But then that second training step there where they give it a "personality"...its just a markov chainer at the end of the day "personality" is fucking with the ideal model. Literally. For fuck s
Re: (Score:2)
It's quite a bit different from a Markov chain, but that is a very useful analogy. I've used the same example before in an attempt to (hopefully) correct some of the mistaken beliefs that people tend to form around programs like this.
Re: (Score:1)
Define a markov chain. Compare.
The problem is when you are starting from the other side, defining a neural network, and then comparing. A markov chain is not a neural network, but a neural network can very easily implement a markov chain, which is exactly what "natural language models" are
The statistical what-comes-next game IS a markov chain. Not just sort-of-like, but actually exactly-like. The fact that the algorithm is _sloppy_ about it, doesn
Re: (Score:1)
You couldn't be more wrong. You seem to have forgotten that in a Markov process the probability of the next state is dependent only on the current state. This is not true for modern language models. RNNs and transformers, for example, are decidedly non-Markovian. RNNs are obvious. Transformers, like GPT, I'll remind you have an attention mechanism.
Hallucinations? (Score:2)
Re: (Score:3)
In the old days we would've called this a prophet and started a religion around it.
I think that's already going on. Microsoft certainly wants you to use the oracle. Google is catching up as fast as they can.
9:22 left in the game... (Score:2)
And Bing is BUSTED!!
things ChatGPT says (Score:2)
It's said a number of wrong things to me over the past week, but one of the funniest was that Hillary Clinton had been President of the United States.
Sometime before 2023 is out, a mom is going to follow medical advice from ChatGPT, and it will result in the death of a child.
Re: things ChatGPT says (Score:2)
And that would be one of those paradoxes of life: an undeniable tragedy for the individual, while a net gain for the species.
Re: (Score:2)
You seem to have forgotten your history...
Re: (Score:2)
The standard for AI tools shouldn't be perfection, and the solution to scenarios like
And if KC Loses (Score:2)
We will have to listen to Mahomes whiny priviledged wife b1tch about how the Eagles targeted her husband's weak ankle and intentionally put him out of the game.
Terminology isn't helping (Score:5, Insightful)
Using terms like "hallucination" to describe GPT text is not helping the public to understand what they're seeing. This is not intelligence. These things are "hallucinating" all their responses, not just the ones that are easily factually checked and determined to be wrong.
Re: (Score:3)
Re: (Score:2)
Are they more copyright protected than any encyclopedia, all of which synthesize older ideas into a new mix of words?
Re: (Score:2)
The model doesn't synthesize ideas though, it synthesizes words. Also known as lossy compression.
And now we know the ultimate truth (Score:2)
Even an AI can't account for an incompetent referee.
Re: (Score:2)
They should have asked it the score of the refball game, instead of the football game.
Dijkstra is hallucinating in his grave (Score:4, Interesting)
I've heard of estimators being described as "smug" when referring to the tendency to favor a wrong estimate with an (erroneously) low uncertainty over one more consistent with reality but far off the current estimate.
Now AIs are "hallucinating."
I'm channelling a spirit. It's coming into view out of the mists of time. It's got a Dutch accent. And it's telling me that there's a special place in hell for people anthropomophize software as an excuse for failing to write correct software.
Re: (Score:3)
They even have theory-of-mind. "Theory of Mind May Have Spontaneously Emerged in Large Language Models" https://arxiv.org/abs/2302.020... [arxiv.org]
Re: (Score:2)
First of all, when a paper says "may have" you can read that as "almost certainly have not".
Second, being able to respond coherently when a user says something that indicates depression or happiness or fear or whatever does not constitute theory of mind. The fact that GPT-3 could pass 70% of their tests indicates that their tests are flawed, not that the GPT-3 has any kind of ToM or sentience.
Re: Dijkstra is hallucinating in his grave (Score:2)
Indeed. I'm developing a test and one of the criteria for whether I'm testing for rote learning or comprehension is to see if ChatGPT can answer the questions correctly. We want a certain amount of testing for obvious basics, but not all that much.
Re: (Score:2)
The fact that GPT-3 could pass 70% of their tests indicates that their tests are flawed, not that the GPT-3 has any kind of ToM or sentience.
True, but that's a better score than I'd expect from the average slashdotter.
Clearly you can hallucinate even if you don't achieve theory of mind.
Re: (Score:2)
an excuse for failing to write correct software.
Is a trained model actually "written?"
not even close to randomness (Score:2)
was it correct? (Score:1)
Superbowl has ended and I still don't know if the hallucination was correct...
Nope. (Score:2)
It failed as I predicted. :)
Shades of CryptoTulips (Score:1)