I don’t think the current common implementation of AI systems are “thinking” and I’ll base my argument on Oxford’s definitions of words. Thinking is defined as “the process of using one’s mind to consider or reason about something”. I’ll ignore the word “mind” and focus on the word “reason”. I don’t think what AIs are doing counts as reasoning as defined by Oxford. Let’s go to that definition: “the power of the mind to think, understand, and form judgments by a process of logic”. I take issue with the assertion that they form judgments. For completeness, but I don’t think it’s definition is particularly relevant here, a judgment is: “the ability to make considered decisions or come to sensible conclusions”.
I think when you ask an LLM how many 'r’s there are in Strawberry and questions along this line you can see they can’t form judgments. These basic but obscure questions are where you see that the ability to form judgements isn’t there. I would also add that if you “form judgments” you probably don’t need to be reminded you formed a judgment immediately after forming one. Like if I ask an LLM a question, and it provides an answer, I can convince it that it was wrong whether or not I’m making junk up or not. I can tell it it made a mistake and it will blindly change it’s answer whether it made a mistake or not. That also doesn’t feel like it’s able to reason or make judgments.
This is where all the hype falls flat for me. It feels like sometimes it looks like a concrete wall, but occasionally that concrete wall is made of wet paper. You can see how impressive the tool is and how paper thin it is at the same time. It’s cool, it’s useful, it’s fake, and that’s ok. Just be aware of what the tool is.
If the LLM could reason, shouldn’t it be able to say “my token training prevents me from understanding the question as asked. I don’t know how many 'r’s there are in Strawberry, and I don’t have a means of finding that answer”? Or at least something similar right? If I asked you what some word in a language you didn’t know, you should be able to say “I don’t know that word or language”. You may be able to give me all sorts of reasons why you don’t know it, and that’s all fine. But you would be aware that you don’t know and would be able to say “I don’t know”.
If I understand you correctly, you’re saying the LLM gets it wrong because it doesn’t know or understand that words are built from letters because all it knows are tokens. I’m saying that’s fine, but it should be able to reason that it doesn’t know the answer, and say that. I assert that it doesn’t know that it doesn’t know what letters are, because it is incapable of coming to that judgement about its own knowledge and limitations.
Being able to say what you know and what you don’t know are critical to being able to solve logic problems. Knowing which information is missing and can be derived from known things, and which cannot be derived is key to problem solving based on reason. I still assert that LLMs cannot reason.