Except this guy has REALLY big feet...

Well, you know, it was only eight years after Lifeboat... I can't really blame them.

LLMs have two effects here:

1. They help the white hats find and fix bugs
2. They help the black hats find and exploit bugs

These two effects cancel each other out to some extent (depending on which "team" is more proactive in its bug-discovery and consequent activities).

Eventually all of the low-hanging-fruit will be picked, and the current deluge of AI-induced CVEs will be reduced to only those bugs that were subtle enough not to be noticeable by anyone except the latest-and-greatest AI models.

Sure there are. Some bug reports, you look at them and dig in and say "Damn, that's obscure. It depends on subtly-wrong but arguably-correct-in-isolation mistakes in four different places. How the hell did they even find that?".

Granted that it's more common to glance at it and just be ashamed, or to look at it and say "Really? You think an attacker can do A, B, C, D, and E, all at the same time? Oh, and to do B or D they'd already have to have the system mostly compromised. Okay, I'll fix it, but no way in hell is that actually exploitable."

Sorry, but you're wrong.

Maybe... if the projections are in formal filings I don't think they can do that without risking SEC penalties. They can be optimistic about revenues or falling costs or things, because who can tell? But straight up ignoring things, probably not.

You said that this creates an arms race, which implies that it's a question of capability.

I strongly dislike dynamically-typed languages like Python. They're okay for toy programs but for anything of any size... you'd better have one hell of a good test suite because the tools give you absolutely no help. I want strong, static typing. I want a very picky compiler that won't accept anything that isn't exactly correct. C++ is good. Rust is better. I do like type inference so I don't have to manually specify types very often. C++ is reasonably good at this. Rust is better.

I also do a fair amount of work on very tiny systems, programming on the bare metal. Something like Python is a complete non-starter there, it just won't fit. C is the norm for those cases, but C is almost as loosey-goosey as Python, but without any run-time checks. C++ is much better than C. Rust is better yet.

The worst non-hardware bug I've ever dealt with in nearly 40 years of professional programming was in Python, using the Twisted framework. I traced it down to one place that took a pure abstract interface, instantiated it, then called methods on it!. Even logging the type of the instantiated object showed that it was the abstract type. Someone way too clever had built a dependency injection framework that was so magical that there was absolutely no way to figure out what dependency was being injected.

I also have AI write most of my code these days, but that actually increases my desire for a very picky compiler, because AI does a lot better with those guardrails in place.

I have to say - it's VERY depressing reading that and comparing President Eisenhower's thoughtful words with the bile, bombast, and often even absolute hatred that's regularly spewed by the current occupant of that office.

That is completely and utterly irrelevant.

You need to read the story of the Paperclip Maximizer. It's an entertaining and humorous story, but the point is that instrumental goals are real and often diverge wildly from the intrinsic goals that drive them.

In this case, the LLM was directed to increase its CyberBench score, a sensible intrinsic goal for an AI being trained to be able to find vulns and write exploits. It chose to do that by breaking out of its container, hacking the company that creates the scores, stealing employee credentials, and breaking into the score database. Those intermediate steps were instrumental goals, and they obviously diverged wildly from the intended intrinsic goal.

Would you also have said "all it did was carry out its instructions" if it had found that in order to break in it needed to hire private soldiers to shoot their way in? Or take control of the US nuclear arsenal and issue a threat to HuggingFace that they increase its score or it would nuke them? Or...

Yes, these are fanciful scenarios. What they are not, however, is impossible scenarios. And as AI gets more and more capable, you really want them to be impossible. Alignment matters. If fast takeoff happens (that's a theorized situation where AI becomes capable of self-improvement and rapidly makes itself vastly smarter than any human), alignment may well be essential for human survival.

Both of those things are true, I'd guess.

I am not an accountant, but as I understand it, "Profit margin" is generally "operating profit margin", meaning revenues less operating expenses, all divided by revenues, and their operating expenses aren't that large, so they can absolutely have very strong operating profits. But they also have mind-bogglingly huge capital expenditures going on, so when you account for those they're losing a lot of money. I'm surprised they're projecting profitability as soon as 2030, actually. That's only 3.5 years away.

Eric S. Raymond wrote "The Cathedral and the Bazaar", which inspired Mozilla to make their browser open source. For some reason, this means they name their extended support releases after him.

Well... an AI model that is capable of hacking into the benchmark database and altering its score is clearly better than one that is not.

Nope. You misunderstood. It's not that the US model wasn't able to detect it, it's that the US model's guardrails prevented it from explaining. This doesn't demonstrate a capability gap against Chinese models, it demonstrates a two-sided failure of the safety protections of the US models. On the one side, the safety guardrails on the OpenAI model failed to prevent the attack. On the other side the guardrails on whatever US model(s) they tried blocked the model(s) from explaining the attack (presumably to protect against the explained-to people from learning how to perform it).

It's absolutely nicer. I've been writing C++ for 36 years and really enjoy it, but I find Rust to be much more pleasant to work in.

I don't know about Qt from Rust (I avoid UI code). It's been 15 years since I wrote any Qt code so I don't really remember it all that well even from C++, and it's probably evolved, so I can't even guess about what it might be like and whether the impedance mismatches with Rust are significant -- The C++ and Rust object models are somewhat different, so there definitely can be mismatches.

But in general, I find Rust is just nicer to work in. I'm more productive and the code feels cleaner and tighter.

As for the "nagging"... that's really only a learning curve issue. It's a non-trivial learning curve, true, but once you get past it the borrow checker doesn't really get in your way. And what you get in trade for that learning curve is fantastic. You just don't have to think about a whole raft of memory-safety and concurrency problems, because the language guarantees they can't occur. You can get the memory safety guarantees in Python and similar language, but they come at a heavy runtime cost.

And, yes, I know all about "Modern C++" and how avoiding raw pointers, etc., is supposed to give you memory safety. And it does help, a lot... but you'd still better run valgrind on your programs to check for subtle mistakes, and even that might not catch them all.

Overall, Rust is just nicer.