1) it's expensive
2) the false-positive rate is very high
I do it for free on my own computer. The lack of signal rendered AI bug hunting pointless having attempted it continuously over the last three years. With current models (e.g. GLM-5.3 754B) both quality of findings and volume of nonsense have in my experience been quite reasonable. I am willing to suffer way worse for the types of results AI has already provided.
This reminds me of the early days of valgrind where it was almost unusable at first and a few years later improved to the point you take literally everything it says seriously. Obviously AI isn't that but the step change is similar.
In the past if you gave models a focus or enough leading questions they might figure it out but never when left to itself. Seems a lot of the recent gains are a result of post training behaviors to act more systematically /w commensurate willingness to burn enormous volumes of tokens rather than any fundamentally new magic. In my case it is a tradeoff between waiting minutes for a worthless answer or coming back the next day for a stack of sometimes amazing results.
That all being said, you're not wrong.
Each LLM is going to be better or worse at finding different bad patterns, and incorrectly identifying certain patterns as bad that are fine.
But the fact is- if you let that LLM throw its head against the wall for long enough- if you burn enough tokens- there's a good chance there will be a couple of diamonds in the sea of bullshit- and that's the problem.
The first time I run code through the model the top line is almost always useful. You have to scroll down to get to the bullshit. Even rerunning the same model against the same code is useful.