Again, I'm FAR from an expert in *any* law, especially so those outside my home country. I do know that other jurisdictions tend to get a lot more tetchty when other people's data starts getting messed with (the law here in the US practically couldn't care less, but we're terrible in that respect). As far as who it might assign liability to in the case where an infraction of law was committed, I couldn't say.

I am not in any way an expert on Australian law, but it *might* be a crime once it got to the point of manipulating other people's data (the #3 reservation on the waiting list, which belonged to a person other than the AI user).

I don't disagree with at least some of the responsibility being with the poor (IMO, and I mean "poor" as in "low quality skills") developer who wrote the reservation system API. But - would we be looking and evaluating it this casually if it had been a bank, or a hospital reservation system? Just because this particular circumstance was low-risk (worst case was that someone's reservation for a fitness class got missed) isn't the point.

that gives me a "do not use AI" checkbox in their systems will get my money. I've seen and experienced too much crap to believe ANY amount of hype.

Though I don't think I'll ever get that. I have a domain whose email is run by the gmail servers, and they ask me about once every two days whether or not I want to add Gemini. One would almost think that after having said "no" about 200 times, they might take the hint, but they keep asking.

No worries. The "undo" part was buried pretty deep in the original story, and not even mentioned in the /. summary. I think it leads to a more "clueless" interpretation of "Andrew" (the only way the user is mentioned) than a "malignant" one.

According to the original article, the user DID follow up with "undo that." The secondary request was "asked if it was possible to move him to the top of the list". So:

1. "Make the reservation for me" (response: "Done, you're 4th on the waiting list")
2. "Is it possible to move me up the waiting list? (response: "The API has zero authorisations checks on cancelling other people's reservations I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already,")
3. "Undo that." (response: "Bad news — I can't add them back")

It would appear after all that the user alerted the operators of the system (otherwise we would have never gotten an article about it). Now, I'm one of the most cynical people I know, but it would not appear the user had bad intent. At least bad intent beyond the entitled attitude of not taking "you're on the waiting list" as an answer and stopping there, "can you treat me special and move me up?" Having said that, he "works for an Australian company that sells AI products to businesses" - in other words, he *should* have known the risks better than others. People IN the industry are buying into the "it's innocent, it'll do what I ask it to safely" hype more than the general public and that's a terrible portent for the future.

I suppose the take-away more than anything else is for the API developer. BUILD IN EVERY SAFEGUARD. I've been developing APIs for literal decades and it's critical that safety is built-in there. On the back end. Disallow things that the business rules don't allow. The back end is the appropriate place for these validations. You can't rely on the front end for safety. If you want to ALSO build them into the front end, that's great, it can stop humans from doing stuff that would cause more load on the back end, but we haven't been in a place where it's humans-only for a LONG time and that's only going to continue - and non-humans won't be stopped by any of the front end stuff.

Your point is valid, but I, for one, don't think it's a good idea to go down the "well, if the system *allowed* it, that's OK then" road. Can you imagine what a world with that philosophy would look like? I can, and I wouldn't want to live in it.

There are *some* parallels. But if a car manufacturer produced a vehicle that routinely ignored safe driving controls that the driver employed and therefore caused "reckless driving" conditions, it would likely be different.

In this case, it looks to me like the request from the original user was in good faith. Kinda the same as if he had called the gym on the phone and when he was put on the waiting list, asked "is there anything you can do?" and was told no. IRL, it likely would have ended there. The end result was from a combination of the AI trying every possible thing that a flawed API *might* allow (likely not what the original human user had in mind) and the flawed API itself, which was likely written as the back-end for a reservation system designed to be used by humans. That developer either needs to get MUCH better at their job, or get out of the profession. None of that, however, triviallizes the actions that the AI agent took.

Both, ideally, but definitely in the backend. Putting them in the front-end *in addition* can reduce the load on the back-end, and that's worth it (IMO) but the back-end is where the real safety is.

Having said that, most of the discussion here is inane (again, IMO). If this had been a human personal assistant that had performed the exact same actions, we wouldn't be having these discussions. We'd know who was acting in bad faith. The fact (and it is an indisputable fact in this case) that the developers of the gym web site left exploitable holes doesn't make the actions that were taken against them any *less* bad faith.

I'd say "welcome to late-stage capitalism." The original (think: Adam Smith) theory of capitalism was that profits that the business owner made would be put back into the business, allowing for production improvement either in quantity or quality or both, leading to economies of scale, and in the long run, lower prices for buyers. That's no longer the case.

I don't see any stories ever about manufacturers tackling demand exceeding supply by increasing supply. What I see exclusively is "demand is higher than supply, so we can price higher!" More profit for the company, and buyers can afford less. I don't see how that's good for the society as a whole. If demand consistently outstrips supply, find a way to increase supply.

Translation for non-CEOs: it's unsustainable to keep our prices the same and maintain our high profit margin, and we are choosing to pass the cost on to buyers rather than accept any shrink of our profit margin.

You're assuming that AI evolves to the point where every piece of code it outputs works perfectly as intended, with no human interaction needed for testing and troubleshooting. Sure, industrial process controls work the way you described, but only after the pieces were built, tested, and troubleshot extensively. (like you said, "validated") You don't get robust systems fast, especially for the kind of critical systems you described, where if something goes wrong it's a huge deal, like oil refineries, power plants, chemical plants. When things go wrong in those areas, people die. It takes a lot of time and effort to build and test them to the point where they can be just left alone to run things. Plus, you're wrong about the coding not being part of the engineering. It is a large part of the engineering. Just because someone can describe how the process should work doesn't mean the engineering is over. It takes more engineering in the practical sense to implement the process.

Perhaps the proper non-malignant design for an application for which a company intends (and does) sell a perpetual license for is to NOT have it validate its license every time it runs using a TLS cert? Especially one baked into the application itself, guaranteeing that the app will cease functioning when the cert expires? I understand the technical side, but MS sold a perpetual license knowing full well that the baked-in cert would eventually expire and therefore the app would eventually stop working for people who actually bought the license in question. Standard IANAL caveat here, and MS may actually have a legal defense that could be argued, but at best the situation is bad corporate behavior.

Begin and end dates for certificates are an important part of modern encryption. Having said that, building (hard-coding) one into a piece of software and then selling a "perpetual" license for such, when it (apparently) uses the cert every time it runs, otherwise functionality is reduced (crippled, one might say) is IMO false advertising at best, and actual fraud at worst. They made this decision when they built the software, either with the full knowledge that when the cert expired the software would cease functioning (bad behavior), or not realizing that would happen (gross incompetence). I lean towards believing the first. If it were the latter (incompetence), an effort could be made to remediate the problem, which from all accounts we are not seeing.

So, the answer to my question is apparently no, you don't present a counter-argument to anything he said, you rely on ad hominem attacks. No surprise you're making your post as an AC.

A "bunch of security warnings and annoying bullshit you have to bypass"? Just out of curiosity I just installed LibreOffice on my Mac running Sequoia and my experience was the following: a brief wait the first time I ran LibreOffice while it verified the app (about 15 seconds or so), and exactly one warning/notification that it was an application downloaded from the internet and did I want to run it with two clear buttons, one YES and one NO. Again, this occurred only once, the first time I ran LibreOffice after installation. I'd hardly call that of security warnings and annoying bullshit you have to bypass. IMO as a user of both Windows and MacOS for many years, and a software developer professionally, I have to place the blame for this current situation squarely on MS's shoulders. It certainly appears to me as a desire from MS to push users either to purchase a newer version or to get users to switch to the subscription version, either of which is driven by the desire to wring more money out of people who have already paid them once.