As long as there’s supervision during training, which there always will be, this isn’t really a problem. This just shows how bad it can get if you just train on generated stuff.
It sounds a lot like this quote from Andrej Karpathy :
Turns out that LLMs learn a lot better and faster from educational
content as well. This is partly because the average Common Crawl article
(internet pages) is not of very high value and distracts the training,
packing in too much irrelevant information. The average webpage on the
internet is so random and terrible it's not even clear how prior LLMs
learn anything at all.
It's already happening. A quote from Andrej Karpathy :
Turns out that LLMs learn a lot better and faster from educational
content as well. This is partly because the average Common Crawl article
(internet pages) is not of very high value and distracts the training,
packing in too much irrelevant information. The average webpage on the
internet is so random and terrible it's not even clear how prior LLMs
learn anything at all.
You have to pretty much intentionally give it enough synthetic data to wreck it. OpenAI and Anthropic train their models on generated data to improve them. As long as there's supervision during training, which there always will be, this isn't really a problem.
The thing about Genshin is that their whole world is set up around nations, and the next one is supposed to be Latin American and West African influenced, but everyone is white. Just like they did the last one, Sumeru, which was Near East and South East Asian inspired. They want the culture with none of the melanin.
Reminder that the author of Glaze, Ben Zhao, a University of Chicago professor stole open source code to make a closed source tool that only targets open source models. Glaze never even worked on Microsoft, Midjourney, or OpenAI's models.
So Kadokawa had this failing mobile game named Kemono Friends, but they wanted a TV anime for it, so they get Omoto Tatsuki to direct it. He takes their product and actually turns it into something people want to watch, going as far as, releasing supplemental clips from the show on Twitter and even arranging a collaboration with the Tobu Zoo in Saitama Japan. This collab gave us the tragic saga of Grape-kun the Humbolt Penguin, a meme in its own right. Since the show didn't have much of a budget he hired no-name voice talent, launching their carriers thanks to the popularity of the show.
Ultimately, it was the clips on Twitter that did him in if rumors are to be believed. He was sharing clips as usual, but Kadokawa wanted that content for the home video special features, so they fired him abruptly. He already saved Kemono Friends for them, so they cut him loose.
Watching that made me sick.