Comment Re:Multiple issues (Score 1) 138
Second, we think there is no limit to how smart an AI can become. This is not true. This is because when you look at charts vs time, they look exponential - showing how each year the AI not only gets smarter but also gets more smarter than it did last year. Those charts so capability vs time but ignore the cost and hardware increases. In reality these charts are NOT showing AI advancements - they are showing Moore's Law.
AI indexes are measuring capabilities of AI systems not Moore's law. You can say moores law is responsible for enabling hardware industrial base but this doesn't change the nature of the thing being measured.
Because of Moores law, each year we get exponentially better chips. But AI itself is not improving, it is the HARDWARE that is getting better - along with the money we spend on the AI. Hardware improvements affect speed, not capability. AI with better hardware is faster, but it can't really do more or give you better answers.
The more training a model of a given size the better answers it gives. The more compute you can afford the more you can afford to train the model.
The honest truth is that all of AI's improvements in capability - the better answers- are entirely caused by HUMANS. The humans detect a problem - putting elephants in a room when told not to - and fix it. The humans realize that AI gives better answers when told to check it's results - so the AI is told to replace "What is the best political party to vote for" with "What are the problems with my answer to what is the best political party to vote for".
This is like saying everything is caused by god and just as useful. Humans are getting better at training AIs resulting in AIs that are more useful and more capabilities. Majority of a models capability and compute budget takes form of pretraining rather than post training where CoT et el is applied.
Consider how easy it is to write a book that has some of your knowledge, but impossible to write a book that has more knowledge than you have.
Similarly, it is extremely unlikely that a species can create an artificial intelligence that is actually smarter than the original species.
This is conflating knowledge with intelligence. What makes AI useful isn't what they know but rather an ability to generalize and apply their experience to new situations. LLMs for example know way more than any human does and their perplexity scores are at least an order of magnitude better than human scores yet nobody would say they are more intelligent than humans.
How could we tell if we succeeded? If it answers a question we cannot answer - how would we know it is right?
I don't think this is a salient issue. Either you get a useful answer or you don't. If what you ask for isn't checkable and you have no way of ever evaluating the real world performance of the answer by putting it to use in some way then what was the point of asking in the first place?
Third and most important, if we can create a super intelligent AI we will not create a single one of them. Instead we will create hundreds of them. There will be the prototype and the one made that fixes the first mistakes. There will be China, Russia, Japans, America, Germany, one. And Microsoft's, Googles, Amazons, etc.
Yep as time moves forward it gets easier and easier for everyone to create their own AI genies. It is ultimately the enabling knowledge and industrial base that matters not how many compliance boxes are checked or how many people on your red team.
I can respect the rare doomer who advocates for blanket AI bans. This at least has some logic to it. While it is infeasible to detect when people are breaking the rules trillions of dollars in capital flows and large scale access to enabling knowledge can't be hidden.
The typical doomer never advocates for stopping. It is just more of the same bullshit of protectionist regulatory hurdles that stand no chance of preventing either the emergence of AI genies or the granting of wishes to different masters. AI companies have already established themselves as wholly untrustworthy power seeking whores (no offense to actual whores)