Well, github seems to be throwing a fit, but I can say from my experiences:
- Uselessly verbose. One AI pull request I was asked to review was aiming to do a minor adjustment the layout of a singular webui element. The pull request was hundreds of lines of CSS because the LLM just started firing the random bullshit CSS cannon, often repeating itself probably because the operator said "no, it's still messed up" until by some miracle the one change he wanted managed to finally appear, alongside a whole bunch of other crap with side effects that the operator didn't bother to find out about. When getting to the heart of what they *actually* wanted, it was a single line one minute css tweak.
- Missing the glaringly obvious. Had a pull request seeking to adjust behavior to be compatible with newer things in the ecosystem. Ok, great, but adjustments had already been made and released a year ago, the operator had a stale container they had never updated. At no point in the clone/pull/mod/pull request flow did the AI stop and say "oh, it appears equivalent changes have already been made", but instead submitted different ways that were actually functionally broken.
- Operators tend to fire off just tons of requests to many projects. A relatively low traffic project I work with that might have a pull request every couple of months woke up with 50 pull requests from one guy that were opened over the course of an hour. The operator had pointed at the issue tracker (which admittedly had poor issue hygiene, resulting in issues open that should have been closed) and said make a pull request per issue to fix everything. One example was a 15 year old issue asking to change the project to support python 2.4, and since then the project had moved to require python 3.9, but the LLM still submitted patches around the specific examples of python 2.4 incompatibilities, despite it being ugly and also useless since so much more of the codebase was python 3 only. Several issues that had been fixed but not updated in the tracker had a pull request to 'fix' it.
- Fixing issues that weren't an issue. They pull a project and ask llm to do a code review and then submit pull requests based on what the LLM represents as needing changes.
So tons of volume, useless changes, changes with side effects...
The main issue is that CodeGen enthusiasts that were formerly intimidated by code syntax and toolchains think they can finally make an impact. The issue being is that code syntax and toolchains are the least of the challenges associated with good software. So CodeGen can significantly mitigate the tedium of those items, but now you have to contend with people that formerly were filtered by the intimidation.