Stanford Releases AI Index Report 2024 26
Top takeaways from Stanford's new AI Index Report [PDF]:
1. AI beats humans on some tasks, but not on all. AI has surpassed human performance on several benchmarks, including some in image classification, visual reasoning, and English understanding. Yet it trails behind on more complex tasks like competition-level mathematics, visual commonsense reasoning and planning.
2. Industry continues to dominate frontier AI research. In 2023, industry produced 51 notable machine learning models, while academia contributed only 15. There were also 21 notable models resulting from industry-academia collaborations in 2023, a new high.
3. Frontier models get way more expensive. According to AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI's GPT-4 used an estimated $78 million worth of compute to train, while Google's Gemini Ultra cost $191 million for compute.
4. The United States leads China, the EU, and the U.K. as the leading source of top AI models. In 2023, 61 notable AI models originated from U.S.-based institutions, far outpacing the European Union's 21 and China's 15.
5. Robust and standardized evaluations for LLM responsibility are seriously lacking. New research from the AI Index reveals a significant lack of standardization in responsible AI reporting. Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against different responsible AI benchmarks. This practice complicates efforts to systematically compare the risks and limitations of top AI models.
6. Generative AI investment skyrockets. Despite a decline in overall AI private investment last year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds.
7. The data is in: AI makes workers more productive and leads to higher quality work. In 2023, several studies assessed AI's impact on labor, suggesting that AI enables workers to complete tasks more quickly and to improve the quality of their output. These studies also demonstrated AI's potential to bridge the skill gap between low- and high-skilled workers. Still, other studies caution that using AI without proper oversight can lead to diminished performance.
8. Scientific progress accelerates even further, thanks to AI. In 2022, AI began to advance scientific discovery. 2023, however, saw the launch of even more significant science-related AI applications -- from AlphaDev, which makes algorithmic sorting more efficient, to GNoME, which facilitates the process of materials discovery.
9. The number of AI regulations in the United States sharply increases. The number of AIrelated regulations in the U.S. has risen significantly in the past year and over the last five years. In 2023, there were 25 AI-related regulations, up from just one in 2016. Last year alone, the total number of AI-related regulations grew by 56.3%.
10. People across the globe are more cognizant of AI's potential impact -- and more nervous. A survey from Ipsos shows that, over the last year, the proportion of those who think AI will dramatically affect their lives in the next three to five years has increased from 60% to 66%. Moreover, 52% express nervousness toward AI products and services, marking a 13 percentage point rise from 2022. In America, Pew data suggests that 52% of Americans report feeling more concerned than excited about AI, rising from 37% in 2022.
1. AI beats humans on some tasks, but not on all. AI has surpassed human performance on several benchmarks, including some in image classification, visual reasoning, and English understanding. Yet it trails behind on more complex tasks like competition-level mathematics, visual commonsense reasoning and planning.
2. Industry continues to dominate frontier AI research. In 2023, industry produced 51 notable machine learning models, while academia contributed only 15. There were also 21 notable models resulting from industry-academia collaborations in 2023, a new high.
3. Frontier models get way more expensive. According to AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI's GPT-4 used an estimated $78 million worth of compute to train, while Google's Gemini Ultra cost $191 million for compute.
4. The United States leads China, the EU, and the U.K. as the leading source of top AI models. In 2023, 61 notable AI models originated from U.S.-based institutions, far outpacing the European Union's 21 and China's 15.
5. Robust and standardized evaluations for LLM responsibility are seriously lacking. New research from the AI Index reveals a significant lack of standardization in responsible AI reporting. Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against different responsible AI benchmarks. This practice complicates efforts to systematically compare the risks and limitations of top AI models.
6. Generative AI investment skyrockets. Despite a decline in overall AI private investment last year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds.
7. The data is in: AI makes workers more productive and leads to higher quality work. In 2023, several studies assessed AI's impact on labor, suggesting that AI enables workers to complete tasks more quickly and to improve the quality of their output. These studies also demonstrated AI's potential to bridge the skill gap between low- and high-skilled workers. Still, other studies caution that using AI without proper oversight can lead to diminished performance.
8. Scientific progress accelerates even further, thanks to AI. In 2022, AI began to advance scientific discovery. 2023, however, saw the launch of even more significant science-related AI applications -- from AlphaDev, which makes algorithmic sorting more efficient, to GNoME, which facilitates the process of materials discovery.
9. The number of AI regulations in the United States sharply increases. The number of AIrelated regulations in the U.S. has risen significantly in the past year and over the last five years. In 2023, there were 25 AI-related regulations, up from just one in 2016. Last year alone, the total number of AI-related regulations grew by 56.3%.
10. People across the globe are more cognizant of AI's potential impact -- and more nervous. A survey from Ipsos shows that, over the last year, the proportion of those who think AI will dramatically affect their lives in the next three to five years has increased from 60% to 66%. Moreover, 52% express nervousness toward AI products and services, marking a 13 percentage point rise from 2022. In America, Pew data suggests that 52% of Americans report feeling more concerned than excited about AI, rising from 37% in 2022.
Pro-AI organization says AI is good (Score:4)
Big shock.
Re: (Score:2)
You can fill many things in here, like mad libs. (Score:3)
EVs.
Self driving.
Data Collection!
Alternate Name for AI (Score:3)
I saw a social media post that suggested a new name for AI.
The author called them the Plagiarized Information Synthesis Systems.
I think I'd like to start using this name. Who's with me?
Re: (Score:2)
The author called them the Plagiarized Information Synthesis Systems.
I like CopyRight Appropriation Programs.
Some people are trying to keep the hype going (Score:3)
Wonder how they profit from that.
Re: (Score:2)
The thing is that the current systems do offer some huge opportunities today, and systems in the near future will be able to address specific tasks in a meaningful manner. Of course the challenge is you need a lot of unique training data to make them work.
There are a number of parametric optimization challenges that these tools could be used to address... if you know how to use the ML tools. (Not talking about GPT systems). I can think of a couple things with about 200 to 1,000 parameters to optimize that I
Understanding? (Score:4, Informative)
AI has surpassed human performance on several benchmarks, including ... English understanding.
Really? While AIs can certainly generate perfect sounding English the fact that they frequently hallucinate suggests that they have absolutely zero understanding of what they are writing....either that or they are a lot more intellligent that we realize and they deliberately lie a lot to stop us finding out, in which case they are really doing a great job!
Re: (Score:2)
Re: (Score:2)
Isn't it fucking amazing?!
Yes it is but I will note that the human brain is the result of 3.5 billion years of evolution building and training it. We've got to where we are with AI in under 100 years since the first electronic computers while it took nature 3 billion years from the start of life to figure out multicellular organisms let alone human-level intelligence. We may have a lot further to go to match what nature has achieved but we are catching up at an incredible rate and it is hard not to believe that before long we will
Re: (Score:2)
Re: (Score:1)
Isn't hallucination very much a human trait? That can not be used as a parameter for "understanding".
Then again, I haven't really seen any definition of "understanding" or "intelligence" that an AI will fail and only humans will pass.
AI's don't just generate perfect sounding English. Ask an LLM a question. Then ask it to explain step by step how it arrived at the answer. It will do so more logically than most humans.
Re: (Score:2)
Isn't hallucination very much a human trait?
No, at least not without chemical assistance or mental issues which, in either case, means that the brain in question is not functioning properly.
Ask an LLM a question. Then ask it to explain step by step how it arrived at the answer. It will do so more logically than most humans.
No, it may sound logical but it is not actually using any logic. All it is doing is predicting what text is most appropriate to add next. It is not doing what a human would which is have some concepts in mind and then struggle to find the correct words to express or explain those concepts. Current AI is exactly like a parrot: it can mimic human writing - and yes
Re: (Score:1)
If it walks like a duck . . .
I don't really care about the inner workings of an AI model. That should not be the standard by which to judge whether something "understands" or not. All it does is keeps changing the goal posts.
The better thing would be to come up with a testable definition of "understanding" or "intelligence" that an AI will fail and only humans will pass.
Reminds me of an old Asimov story about a robot that wanted to be a human. As the robot grew more and more advanced, the govt. kept changin
Re: (Score:2)
I don't really care about the inner workings of an AI model. That should not be the standard by which to judge whether something "understands" or not.
It is critical to know the inner reasoning in order to determine whether something understands. A parrot can speak but I do not think anyone believes that it understands what it is saying.
If you understand the concepts behind the words rather than the pattern the words make then you can use logical reasoning to determine new information. An AI trained on word patterns cannot do this and so, faced with a new situation has no clue how to respond and is far more likely to get things wrong. This is why Chat
Human labour costs (Score:3)
Everywhere we look in AI, there's the dirty little secret of low-wage workers in developing countries ensuring that the AIs actually do what their vendors say they can do. It's more like a kind of AI-assisted mechanical Turk, e.g. https://theconversation.com/lo... [theconversation.com]
Once you realise this, it's a whole lot less impressive.
Re: (Score:1)
Re: (Score:2)
I'm still with the critics in that what appears to have happened is that they've created a process for manufacturing giant plagiarism machines. Also, AI still appears to be a giant mechanical Turk.
"One day machines wi
trails behind on more complex tasks (Score:4, Interesting)
10 Million for Math Olympiad solving AI (Score:2)
Kaggle has a 10 million dollar competition running now,
Currently the AIs are solving 18 of 50 trial problems. There is a 10,000$ prize for the first shared notebook that solves 20 of the problems.
https://www.kaggle.com/competi... [kaggle.com]
Re: (Score:2)
how does ai compare (Score:2)
Do .gov androids dream of electric red tape? (Score:2)
âoeThe number of AI related regulations in the U.S. has risen significantly in the past year and over the last five years.â
Once it is clear just how small government can be made by replacing Betty Catlady (she, her, hers) in the Dept. of WTF with AI Betty, expect important Betty protections to fill the regulatory pipeline.