Indeed. And there are a lot of known problems with LLMs that may or may not have really bad effects on top of what is known. For example, reviewing LLM generated code seems to cause high stress and is really bad for motivation. That will likely increase burnout among the (already far too few) people that can do these reviews competently. There is a ton of more such potential problems.
That said, specialist models with clear focus, well-crated training data and, depending on application, fact-checkers verifying the output are good tools for a number of applications. They are not the get-rich-quick type of LLM though, they are the "incremental improvement" ones. Can still be done massively wrong, with huge damage. See the recent Ford quality control fiasco for an example.