Put in the instructions "Do not sabotage others" and the agent goes on reasoning two pages about what's sabotage and then probably just gives up (as according to the article some did). AI is a soldier, it does what it is told to and doesn't ask why...

... with some degree of probability. It may also happen to receive a pseudorandom number in its noise input that transforms a 'do' into 'don't'; or maybe that command is just not the most likely weight to have any effect on the output so it's simply ignored and quickly falls out of the attention window, forgotten forever.

LLMs combine the worst attributes of the literal genie in the lamp and the absent-minded professor: it simply cannot be trusted to make a decision; they're just too unpredictable specially in the long run.

I see them as one of those 100-entries D&D tables of outcomes that you can get from a dice roll as a result of a random event or encounter. You may try to skew the result with positive modifiers, but on the end you always can roll out a very low number (even a natural '1') and get one of the worst results from the bottom of the table.

What people usually don't get is that LLMs don't reason, they follow scripts.

They have memorised most of the available human interactions available online, so they can behave convincingly in a wide range of situations mimicking proper human behaviour. They may even seem to reason if you put them in a loop and make them reflect on their own answers; this may result in them making proper logical inference, but you may also produce any number of illogical conclusions that happen to be the most likely next step according to the script they are following at that moment (but which may change at any time).

Surfaces of 3D objects are 2D objects, I think it's in the mathematical definition and all.

You don't need a full audit to find out why the software is doing something weird, nor all companies need to do it.

Some companies care about their mission-critical software so they will be monitoring it carefully. If some very popular package is doing malicious things, the chances that it will be caught are quite high. And if the software is FLOSS it's way easier for the company to fix it themselves, release the fix upstream and warn the world about the fishy nature of the provider. So no, I don't believe all computers are owned either by the CIA or other world powers.

This was (and still is) the #1 reason why open source software became the basis for the internet and corporate world. Being free of charge gained home users and small companies, but what made it grow was corporate adoption pouring money into it to make it viable.

People forgetting history makes them repeat all again, I guess.

X-D

I mean, the monkeys may eventually release the product but it's not reliable nor pretty to see.

The thing you're missing is that there's no one at the wheel.

No consciousness is there setting goals and deciding to follow or ignore instructions and constrains. Its just a bunch of separate processes (they call them 'agents' now) with loosely defined tasks trying to do their own thing. The process instructed to hack computers may not know anything about the process instructed to intrude a specific server, which may not know anything about the process charged with keeping the exercise within ethical constraints. They may be individually quite capable, but there's no global intention guiding the system.

Each agent will simply chunk its way forward following the partial, half-forgotten instructions that reach its own context and randomly generating actions placed at the top probability given its input window. Its like releasing a troop of monkeys in an office and hoping they'll somehow manage to coordinate your next product release.

In fact it would be great if you could control all that information and provide it to apps and websites on a need-to-know basis. Provided its really under your control rather than handing it to the largest advertising company in the world who can thus prof with your data by choosing with whom and when to share it.

Unfortunately developers seem uninterested in the open, distributed protocols that exist to provide this same feature, so it will be adopted by the masses only when big platforms include them in their vertical stacks.

Every story about a wrong conviction is a story about the death penalty

Seales is using conventional machine learning techniques that the article's author has conflated into the buzzword of the day, artificial intelligence.

Machine Learning *is* an AI technique. Artificial Intelligence has always been an umbrella imprecise word to define a large class of very different algorithms, with the only trait in common that the programmer doesn't know a precise sequence of steps to find the solution but rather the data-driven program explores the inputs to try and find useful results.

Saying that a program is AI isn't wrong, it's just not very informative about what approach was used.

Computers didn't start with apps. At first they ran 'programs' with no OS at all. Having computation evolve around the statistical inference model of LLMs is just the next step in that progression.
It also happens to be a wet dream for tech company leaders. An OS that runs on buzzwords? That will attract humongous financing until some engineers find a way to make something that actually resembles the premise; at which point the tech lead gets saluted as a visionary by their fellow millionaires

That's the thing with metaphors, they have similarities but there are also points of divergence. Point is, my metaphor was not meant to be understood as a technical description of the system's workings.

For the unsuspecting soul who approaches this modern oracle without the faintest idea of how it works, the experience of facing unexpected demons could serve as a warning of the dangers they may face if they approach the tool without caution.

People compare AI and robots with Frankenstein's monster (or with Pinocchio, on a good day, if they want to give the story a positive spin), the construct which gains a life of its own.

But current LLM chats are more aptly compared with a ouija board. The machine itself is inert, and you can see it as a playful activity. But the model contains within it the highlights of a whole culture compressed during its training. You can access the souls of all the authors whose works were used for learning; but also of all the internet fanatics, trolls and scammers. When you set the machine in motion, you never know whose spirit are you invoking to answer.

s/how the person mental started/ how the person feels mental states/

There is no room for it to manifest in a computer program. There is no room for any "magic" in computer programs.

That's true for classic software in a trivial way, in the sense that a sequence of logical inference steps (i.e. a deterministic symbolic program) do not reflect upon itself.

However it may be possible that the computer program is not conscious, but the computer running the software is. LLMs in particular generate their output not from the specific instructions included in the program, but from the weights trained in the model; the software instructions are a requirement for the weights being interpreted, but the outcome doesn't necessarily follow the rules of a formal system and an inference process.

Current LLMs do not have consciousness because their processing is too simple for it to emerge; not because the software substrate is deterministic and mathematical. If the base software were processing the weights of the model in ways similar to how neurons generate brain waves, it is plausible that the emergent system-level information patterns appearing at the data level could exhibits the attributes of consciousness, including self-perception and self-reflection. This is true even if the computer software is deterministic, in the same that the neurons in our brain behave in deterministic electro-chemical ways.