You're spot on regarding how AI operates.

AI is stupid story time!

I recently helped a friend with a self-hosted VPN problem. He had been using a free trial of Gemini Pro to try to fix it himself but gave up after THREE HOURS. It never tried to help him diagnose the issue, but instead kept coming up with elaborate fixes with names that suggested they were known issues, like The MTU Traffic Jam, The Packet Collision Quandary, and, my favorite, The Alpine Ridge Controller Trap. Then it would run him through an equally elaborate "fix". When that didn't work, it would use the failure conditions to propose a new, very serious sounding pile of bullshit and the process would repeat.

I fixed it in about fifteen minutes, most of that time spent undoing all the unnecessary static routing, port forwarding, and driver rollbacks it had him do. The solution? He had a typo in the port number in his peer config.

I can't deny that LLMs are full of useful knowledge. I read through its output and all of its suggestions absolutely would have quickly and efficiently fixed their accompanying issue, even the thunderbolt/pcie bridging issue, if the real problem had been any of them. They're just garbage at applying that information.