16kloc is a small project.

I should soften what I said before: You can get away with seat-of-the-pants work if one or more of: (a) your projects are small, (b) your projects are non-critical or (c) you invest in massive testing and validation.

In my old job, if I made one kind of mistake it could prevent three billion Android devices from being able to boot. If I made a different kind, it could result in the same number of devices being insecure against data extraction and other attacks (of course, OEMs often made those mistakes for me). In my new job, if I make mistakes vehicles can crash and people can die.

I want all the help the tooling can give me, and I spend a fair amount of my time building additional tools to prevent mistakes. None of it eliminates the need for testing, of course, especially when lives are on the line, but bugs that get caught by the compiler cost almost nothing. Bugs that get caught by QA, or, far worse, in the field, are very expensive.

Note that there are big systems built in Python. Instagram has millions of lines of Python code. They also mandate explicit type declarations everywhere and they built their own static typechecker, Pyre. Lots of other people have built typechecking libraries and tools for Python, trying to fix that problem. Maybe the next time I write Python I should try some of those tools.

Though I probably won't ever write any Python again. I really only use it for quick and dirty scripts, and these days I tell AI to write those for me, and I don't even bother to read the code. If it works, fine. If it doesn't, I tell the LLM to figure out why, and fix it.

Except this guy has REALLY big feet...

Well, you know, it was only eight years after Lifeboat... I can't really blame them.

LLMs have two effects here:

1. They help the white hats find and fix bugs
2. They help the black hats find and exploit bugs

These two effects cancel each other out to some extent (depending on which "team" is more proactive in its bug-discovery and consequent activities).

Eventually all of the low-hanging-fruit will be picked, and the current deluge of AI-induced CVEs will be reduced to only those bugs that were subtle enough not to be noticeable by anyone except the latest-and-greatest AI models.

Sure there are. Some bug reports, you look at them and dig in and say "Damn, that's obscure. It depends on subtly-wrong but arguably-correct-in-isolation mistakes in four different places. How the hell did they even find that?".

Granted that it's more common to glance at it and just be ashamed, or to look at it and say "Really? You think an attacker can do A, B, C, D, and E, all at the same time? Oh, and to do B or D they'd already have to have the system mostly compromised. Okay, I'll fix it, but no way in hell is that actually exploitable."

Sorry, but you're wrong.

Maybe... if the projections are in formal filings I don't think they can do that without risking SEC penalties. They can be optimistic about revenues or falling costs or things, because who can tell? But straight up ignoring things, probably not.

You said that this creates an arms race, which implies that it's a question of capability.

I strongly dislike dynamically-typed languages like Python. They're okay for toy programs but for anything of any size... you'd better have one hell of a good test suite because the tools give you absolutely no help. I want strong, static typing. I want a very picky compiler that won't accept anything that isn't exactly correct. C++ is good. Rust is better. I do like type inference so I don't have to manually specify types very often. C++ is reasonably good at this. Rust is better.

I also do a fair amount of work on very tiny systems, programming on the bare metal. Something like Python is a complete non-starter there, it just won't fit. C is the norm for those cases, but C is almost as loosey-goosey as Python, but without any run-time checks. C++ is much better than C. Rust is better yet.

The worst non-hardware bug I've ever dealt with in nearly 40 years of professional programming was in Python, using the Twisted framework. I traced it down to one place that took a pure abstract interface, instantiated it, then called methods on it!. Even logging the type of the instantiated object showed that it was the abstract type. Someone way too clever had built a dependency injection framework that was so magical that there was absolutely no way to figure out what dependency was being injected.

I also have AI write most of my code these days, but that actually increases my desire for a very picky compiler, because AI does a lot better with those guardrails in place.

I have to say - it's VERY depressing reading that and comparing President Eisenhower's thoughtful words with the bile, bombast, and often even absolute hatred that's regularly spewed by the current occupant of that office.

That is completely and utterly irrelevant.

You need to read the story of the Paperclip Maximizer. It's an entertaining and humorous story, but the point is that instrumental goals are real and often diverge wildly from the intrinsic goals that drive them.

In this case, the LLM was directed to increase its CyberBench score, a sensible intrinsic goal for an AI being trained to be able to find vulns and write exploits. It chose to do that by breaking out of its container, hacking the company that creates the scores, stealing employee credentials, and breaking into the score database. Those intermediate steps were instrumental goals, and they obviously diverged wildly from the intended intrinsic goal.

Would you also have said "all it did was carry out its instructions" if it had found that in order to break in it needed to hire private soldiers to shoot their way in? Or take control of the US nuclear arsenal and issue a threat to HuggingFace that they increase its score or it would nuke them? Or...

Yes, these are fanciful scenarios. What they are not, however, is impossible scenarios. And as AI gets more and more capable, you really want them to be impossible. Alignment matters. If fast takeoff happens (that's a theorized situation where AI becomes capable of self-improvement and rapidly makes itself vastly smarter than any human), alignment may well be essential for human survival.

Both of those things are true, I'd guess.

I am not an accountant, but as I understand it, "Profit margin" is generally "operating profit margin", meaning revenues less operating expenses, all divided by revenues, and their operating expenses aren't that large, so they can absolutely have very strong operating profits. But they also have mind-bogglingly huge capital expenditures going on, so when you account for those they're losing a lot of money. I'm surprised they're projecting profitability as soon as 2030, actually. That's only 3.5 years away.

Eric S. Raymond wrote "The Cathedral and the Bazaar", which inspired Mozilla to make their browser open source. For some reason, this means they name their extended support releases after him.

Well... an AI model that is capable of hacking into the benchmark database and altering its score is clearly better than one that is not.

Nope. You misunderstood. It's not that the US model wasn't able to detect it, it's that the US model's guardrails prevented it from explaining. This doesn't demonstrate a capability gap against Chinese models, it demonstrates a two-sided failure of the safety protections of the US models. On the one side, the safety guardrails on the OpenAI model failed to prevent the attack. On the other side the guardrails on whatever US model(s) they tried blocked the model(s) from explaining the attack (presumably to protect against the explained-to people from learning how to perform it).