Comment Re:nothing's impossible (Score 1) 63
It probably has been trained on chemical equations and synthesis. There are millions, if not billions, of completely harmless uses. It doesn't need to be trained specifically on cocaine in order to A) know that cocaine exists, B) figure out the chemical formula to cocaine, C) use standard, generally available science to synthesis the formula.
The LLM has to be trained on what cocaine is and why it shouldn't be offering information about it, otherwise you can't even try to prevent it from outputting cocaine information in the first place.
You can't tell someone "Don't do XYZ, it's illegal", and expect them to be successful, if you never explain to them what XYZ is, or how to identify it.