I think we need to conceptually seperate "synthetic data" from the "model collapse" mentioned above.
Synthetic data does not add more information: it helps solve the problem of overfitting the model to the training data, so that it performs poorly on a separate test data set.
With image classification, it is clear from research that "data augmentation" by zooming, adding noise, streching, rotating, shearing and changing the color space can increase the effective size of your annotated image dataset and thus reduce overfitting and help generalize the data:
https://innotescus.medium.com/...
Another technique is to take your 100 cat imges and paste them randomly over say 1000 random images, genereating tens of thousands of different images.
Synthetic data has several advantages besides reducing overfitting: even up class distributions, reducing bias, and potentially, improving privacy.
https://syntheticus.ai/blog/th...
The big problem is when you use synthetic data generated by a generative model to train the same generative model (rather than a classification model).
https://www.techtarget.com/wha...
You end up with a death spiral of "model collapse" that cannot be recovered from.