Since yours was the first reply I came to that didn’t just ‘react’, I’d like to challenge 1 point in your list (the rest I pretty much agree with), and that is the first one. For context, I worked in AI (or ML as it was known then) in the 1990s. The models were very much based on ideas from neuroscience (my CS PhD supervisor was a biologist). Saying “they can’t think” requires a precise definition of what “thinking” is, and I’ve not seen one so far.
For sure, the most current LLMs are not what we might call human-level sentient, and have only seen a fraction of what a human baby would be exposed to in terms of training data. But as far as the way they process that data, perhaps they are “thinking” in the same way a brain would think if all it ever ‘saw’ was text. Perhaps they think in the same way an insect or small rodent thinks. And as they grow larger / more sophisticated, the same as a dog or cat? Or a small primate? You can see where that’s going.
Anyway, I enjoyed The Infinity Machine by Sebastian Mallaby. My PhD was based on the early work of Yann LeCun, and putting all those names and the motivations behind them into a full picture was eye-opening.
You say that, and GAs were used decades ago to design FPGAs to a spec. The evolved design worked perfectly on the test chip, so the design was copied onto a second chip and it failed. The logic gates were identical but the GA had utilised microscopic differences in the substrate and there were large areas of programmed chip totally unconnected to the main circuit. Without them, the first chip didn’t work any more.
There are likely quantum effects available at the size / scale of neurons, and it’s brave to say evolution wouldn’t exploit them if there was some benefit.