“An intriguing open question is whether the LLM is actually using its internal model of reality to reason about that reality as it solves the robot navigation problem,” says Rinard. “While our results are consistent with the LLM using the model in this way, our experiments are not designed to answer this next question.”
The paper, “Emergent Representations of Program Semantics in Language Models Trained on Programs” can be found here.
Abstract
We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code.
Wow, that headline is significantly overstated. This has nothing to do with “understanding” or “reality”. Like holy shit this is an absurd leap from what they actually state in the abstract. Thanks OP for including it in the “preview” here.
Given a number of “starting states” and “ending states” of 2D grid based “environments”, an LLM is able to return increasing accurate “between states” as it is trained.
Neat! Not an understanding of reality. At the biggest stretch it is an “understanding” of an incredibly constrained problem space.
I did some source digging to hopefully best address your observations. Science journalism (even when internal and likely done in concert with the authors) is fundamentally a game of telephone. But looking at the source papers:
They say it in an incredibly formal way, but they do seem to come to the conclusion that the LLM develops understanding. The paper makes that case within an incredibly narrow context, but it does include:
We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. We argue that the observed semantic content cannot be fully attributed to a retrieval-like process, and instead requires the LM to perform some degree of generalization over the semantics. More broadly, we see programs and their precise formal semantics as a promising direction for working toward a deeper understanding of the behavior of LMs, such as whether or how LMs acquire and use semantic representations of the underlying domain more generally.
With it now clear that the generalized case is not shown: the specific type of understanding that they have shown is non-trivial.
Conclusion: This paper presents empirical evidence that LMs of code can acquire the formal semantics of programs from next token prediction.
A foundational topic in the theory of programming languages, formal semantics (Winskel, 1993) is the study of how to formally specify the meaning of programs.
From Winskel: The Formal Semantics of Programming Languages provides the basic mathematical techniques necessary for those who are beginning a study of the semantics and logics of programming languages. These techniques will allow students to invent, formalize, and justify rules with which to reason about a variety of programming languages.
Also notable but unrelated: Jin and Rinard’s paper was supported, in part, by grants from the U.S. Defense Advanced Research Projects Agency (DARPA).
I mostly get what you’re saying, though I don’t have the requisite understanding to follow formal proofs, but if there is one thing I do know for certain, it’s that “understanding” is anthropomorphizing and shorthand for something that is very much not understanding in a human context at all.
I get that it can be hard to find the right words to explain a some of these emergent phenomena, but I think it’s misleading to use words that make AI appear to have a thought process akin to anything we could understand as such—at least in settings where folks might not understand the shorthand as such.
And maybe everyone here is aware of that, but it makes me uneasy, hence this comment to hopefully make that point.
The paper is kind of saying that as well. I added a quote to the post to help set the context a bit more. As I understand it, they’ve shown that an LLM contains a model of its “world” (training data) and that this model becomes a more meaningful map of that “world” the longer the model is trained. Notably, they haven’t shown that this model is actively employed when the LLM is generating text (robot commands in this case), only that it exists within the neural network and can be probed. And to be clear - its world is so dissimilar from ours, the form its understanding takes is likely to seem alien.
These techniques will allow students
As someone who understands formal proofs, it’s completely misleading to conflate formalism with sketchy pedagogical theories (wtf).
Yes, terminology like “understands” is a choice outside of formalism that’s intentionally misleading for the sake of marketing/funding.
Genuine question: What evidence would make it seem likely to you that an AI “understands”? These papers are coming at an unyielding rate, so these conversations (regardless of the specifics) will continue. Do you have a test or threshold in mind?
We’re all sick of LLM, but the article is actually a really interesting read. How any of these systems can “understand” anything remains to be sufficiently explained. The abstract is not very indicative of the article content.
From MIT again - our exploration of how LLMs can do the things they can is pointing us in some interesting directions re: our exploration of how our own brains understand.
How any of these systems can “understand” anything remains to be sufficiently explained.
It’s not explained at all, but somehow the headline still exists.
deleted by creator
i feel that many of us, when confronted to this, are doing like Z.B. (president of the Galaxy) in The hitchhiker guide… when he says :
… “whenever I stop and think why did I want to do something? – how did I work out how to do it? – I get a very strong desire to just to stop thinking about it” …
We don’t want to be surpassed by machines … and this explains the large amount of downvotesI’m actually pretty sure the downvotes are because LLM’s don’t think. They don’t even process. They pick the highest number and spit out the information attached to it.
Do I “think” or does my brain pick the closest neuron and spit out a function based on that input?
If we could recreate the universe, would I do the exact same thing in the exact same situation?
I’m sorry. Because you don’t understand how your brain works you’re suggesting that it must work in the same way as something a similar brain created because you don’t know how either thing works. That’s not an argument.
No, I’m not suggesting that.
I’m suggesting that if we don’t even understand how consciousness works for ourselves, we cannot make claims about how it will look for other things.
Deterministically free will does not exist, if we cannot exercise free will we cannot have independent thoughts just the same as a machine.
Truth is we don’t really know shit, we’re biological machines that are able to think they’re in control of themselves based on inputs. If we ever discover true AGI it will be on accident as we fiddle with technologies such as LLMs or any other complex models.
Okay. Feed a new species that hasn’t been named yet into an LLM. Does it name that new creature? Can it decide what family or phylum etc it belongs? Does it pick up the specific attributes of that new species?
It might be able to pick those things out, I certainly couldn’t.
Edit: So ChatGPT correctly identified a new species from 4 days ago as a type of Storm Petrel and a new flower from Sri Lanka as an Orchidaceae. Far better than I could do.
That is very deliberately not in the spirit of the question I asked. It’s almost like you’re intent on misunderstanding on purpose just so you can feel like you’re right.
I suspect others are talking about “thinking” only objectively.
B) If a LLM had a subjective experience when given input presumably it has none when all processes are stopped (subjectively, unverifiable).
A) If a LLM has no input then there are no processes going on at all which could be described as thinking (objectively, verifiable: what is the program doing).
I’m suggesting that if we don’t even understand how consciousness works for ourselves, we cannot make claims about how it will look for other things.
So is a rock conscious? I guess we’ll never know… But AI!?! Definitely conscious! smh.
Science cannot say much about what it is to think since it doesn’t understand the brain well enough … and the day we can fully explained it, we will also be able to replicated it on computers.
Science can and does quantify what our brains do vs what an LLM does though. That’s the point. That’s why the brain knows when it’s supplying wrong information or guessing but the LLM does not.
The LLM can provide wrong information. What it can’t do is intentionally lie.
i agree with you that we are much better than LLMs in the fact we can verify our errors (and we can do much more things LLMs don’t do).
Still i am happy to have access to their vast memory and i know where they fail most of times so i can work with them in a productive way.
The day we provide them (or DNNs) with “Will” is i think when they will become (more) dangerous.Gotta use extra heavy quotes around “will”… Mixing up AI pseudo-science with fascist pseudo-science.
Nah fam. You’re comparing this thing to yourself and other humans because to you it sounds like us. “Sounds like” is the phrasing conspiracy theorists use to try to make correlations regardless of whether they make sense or not without filling in any actual facts. Either you know and understand how they work and are researching this and have a valid rebuttal or you don’t. But an assumption does not make anything you said fact. People who are researching this have already refuted this claim.
You don’t even know what the word “will” would mean in this context, or that it would provide an LLM with the kind of consciousness required to be sapient. So can we stop arguing if you just admit that you like using LLM’s so you have a bias and a poor understanding of what they are and aren’t.
Science cannot say much about what it is to think since it doesn’t understand the brain well enough
Wild pseudo-scientific generalization.
and the day we can fully explained it, we will also be able to replicated it on computers.
There are many many things that are fully explained but will never be replicated on computers. Eg. Any numerical problem bigger than a computer.
This is what “high class” fake news looks like.
deleted by creator
