An MIT team used probing classifiers to investigate if language models trained only on next-token prediction can capture the underlying meaning of programming languages. They found that it forms a representation of program semantics to generate correct instructions.

“An intriguing open question is whether the LLM is actually using its internal model of reality to reason about that reality as it solves the robot navigation problem,” says Rinard. “While our results are consistent with the LLM using the model in this way, our experiments are not designed to answer this next question.”

The paper, “Emergent Representations of Program Semantics in Language Models Trained on Programs” can be found here.

Abstract

We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code.

  • wizardbeard@lemmy.dbzer0.comEnglish
    0·
    3 months ago

    Wow, that headline is significantly overstated. This has nothing to do with “understanding” or “reality”. Like holy shit this is an absurd leap from what they actually state in the abstract. Thanks OP for including it in the “preview” here.

    Given a number of “starting states” and “ending states” of 2D grid based “environments”, an LLM is able to return increasing accurate “between states” as it is trained.

    Neat! Not an understanding of reality. At the biggest stretch it is an “understanding” of an incredibly constrained problem space.

    • Hackworth@lemmy.worldOPEnglish
      1·
      3 months ago

      I did some source digging to hopefully best address your observations. Science journalism (even when internal and likely done in concert with the authors) is fundamentally a game of telephone. But looking at the source papers:

      They say it in an incredibly formal way, but they do seem to come to the conclusion that the LLM develops understanding. The paper makes that case within an incredibly narrow context, but it does include:

      We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. We argue that the observed semantic content cannot be fully attributed to a retrieval-like process, and instead requires the LM to perform some degree of generalization over the semantics. More broadly, we see programs and their precise formal semantics as a promising direction for working toward a deeper understanding of the behavior of LMs, such as whether or how LMs acquire and use semantic representations of the underlying domain more generally.

      With it now clear that the generalized case is not shown: the specific type of understanding that they have shown is non-trivial.

      Conclusion: This paper presents empirical evidence that LMs of code can acquire the formal semantics of programs from next token prediction.

      A foundational topic in the theory of programming languages, formal semantics (Winskel, 1993) is the study of how to formally specify the meaning of programs.

      From Winskel: The Formal Semantics of Programming Languages provides the basic mathematical techniques necessary for those who are beginning a study of the semantics and logics of programming languages. These techniques will allow students to invent, formalize, and justify rules with which to reason about a variety of programming languages.

      Also notable but unrelated: Jin and Rinard’s paper was supported, in part, by grants from the U.S. Defense Advanced Research Projects Agency (DARPA).