This has been happening forever. The local LLM folks poke them with riddles all the time, but then they get obviously trained in.
What’s more, standard tests like MMLU are all jokes now. All the major LLMs game the benchmarks and are contaminated up and down; Meta even got caught using a specific finetune to game LM Arena. The only tests worth a damn are those in niche little corners of the internet no one knows about, or niche private ones.

Woke liberal propaganda.
Even my Dad is talking about how this is all stupid, and even if it isn’t, geoengineering is going to fix it. Gah, the world is fucking nuts.