An LLM Isn't a Compiler

Every time there is a debate online about blindly trusting the code an LLM generates, inevitably someone makes the following argument:

If you don’t trust what the machine outputs, why are you using a compiler? You should be writing machine code directly!

With such a statement, they imply that an LLM behaves the same as a compiler. I think that LLMs differ from compilers in some fundamental ways, which is why I can trust compiler output but cannot do the same for LLM generated code. And no, it’s not determinism! It’s that a compiler is supposed to preserve the meaning of a program, while an LLM has to infer what that program should mean.

I also think that even the most enthusiastic vibe coders, deep down, know about this difference, and I will try to prove it at the end of the post.

Determinism Isn’t Enough

To make it easier to reason about the behavior of compilers and LLMs, we’ll model them as mathematical functions. Defining a function for compilers is straightforward:

where $S$ is a set of source programs and $T$ is a set of target programs.

At first glance, it might seem that LLMs can’t be modeled as functions. That’s because a function returns the same output for the same input, and LLMs don’t. We all know that the same prompt can result in different outputs. But, roughly speaking, LLM token generation is made up of two parts:

1. Multiply some matrices to get next token candidates.
2. Select the next token based on some probability distribution.

The first part is just basic maths, so it’s deterministic¹. The second part can also be modeled as deterministic. We can use a pseudorandom number generator (PRNG) when selecting the next token, which allows us to seed the PRNG and get a deterministic function.

With this insight, we can model LLMs as

where

• $P$ is a set of prompts.
• $C$ is a set of contexts. A context includes a system prompt, conversation history, …
• $R$ is a set of random seeds.
• $O$ is a set of outputs.

With LLMs modeled as deterministic functions, I think it’s pretty obvious that determinism is not the thing that makes people trust compiler outputs but not trust LLM generated code. For those of us who still like to look at the code, does exposing a seed parameter of a model we’re calling change anything? If you could put $42$ as a seed and get Claude to output code deterministically, would you trust the code more? Would you stop reviewing it? I am pretty certain that the answer to those questions is No.

Compilers Preserve Meaning

The semantics of a program describe its meaning: the behavior defined by the source code. We’re going to denote the semantics of a program $P$ as $[[P]]$.

In programming language theory, there’s a lot of talk about semantics. One property we really like compilers to have is for them to keep the semantics of the program. The meaning of the compiled output should be the same as the meaning of the source code. In other words, for a compiler $f$, we expect:

In fact, compilers usually have this as a requirement. If the output of a compiler does a different thing than you described in the source code, most compiler developers will treat this as a bug.

For LLMs, there’s no such guarantee. For a prompt $p$ and a context $c$, let’s define $A(p,c)\subseteq O$ as a set of acceptable outputs. That is a set of LLM generated programs that do what we intended. An analogous semantics preservation property would be

which obviously doesn’t hold.

Prompts Are Ambiguous

There is another problem with using LLMs to generate code. You interact with them using natural language. And natural language is messy and ambiguous. We have all heard this joke before:

A programmer’s wife tells him:
“Go to the store and buy a carton of milk. If they have eggs, get a dozen.”

The programmer goes to the store and comes back with 12 cartons of milk.

As much as this is a joke about programmers taking instructions literally, it’s also an amazing example of how messy and ambiguous natural language is. There are two ways to understand the sentence above, and you need a ton of insight about how humans work to understand that your wife wants a box of eggs, not 12 cartons of milk.

We’ve spent centuries developing formal mathematical language. We have an entire field studying formal languages. And there’s a reason for that: ambiguity makes reasoning harder. Just look at this post! The formal math symbols let us state precisely what we mean instead of leaving it to interpretation.

An LLM Is Not a Compiler

All of this finally brings us to why I believe an LLM is not a compiler and never will be. Already, by itself, it doesn’t make any guarantees that the semantics will be preserved. And when you combine that with the ambiguity of natural language, you get unpredictability.

Again, when talking about unpredictability, I don’t mean whether you get the same output twice (determinism). I mean whether you get the output you intended — and whether you even did a good job encoding your intentions in the ambiguous mess called natural language.

And as I alluded to in the beginning, even the workflows of the most prominent vibe coders, who tell you not to look at the code, reflect this.

When you work with a compiler, what do you commit to your repository? Is it the source code or the compiler output? When you want to understand what someone else on your team did, do you look at the source code or the compiler output? When something breaks and you need to fix it, do you look at the source code or the compiler output?

When you vibe code a project, what do you commit to your repository? Is it the prompts or the LLM output? When you want to understand what someone else on your team did, do you look at their prompts or the LLM output²? When something breaks and you need to fix it, do you look at the original prompts or the LLM output?

Challenge

Here’s a challenge for anyone who wants to prove me wrong and earn the right to compare LLMs to compilers. Create at least a medium sized project that doesn’t contain any source code. It should contain only prompts. Its README.md should read something along these lines:

To build this project, open Claude with the model set to
`Sonnet 5.5 High` and give it the following prompt:

```
Read `BUILD.md` and save the generated program to `./out`.
```

The point of this challenge is not just to get an LLM to generate working code every time you run the main prompt. It’s to end up with a project you can iterate on by editing the prompts, where adding one feature doesn’t unexpectedly change how the rest of the program behaves.

Conclusion

Because compilers use formal language as input and try really hard to preserve the meaning, they’re predictable. They’re predictable in a way that lets you store only the source code because it’s enough to tell the whole story of the software.

LLMs, on the other hand, deal with the ambiguity of natural language and don’t guarantee the meaning will be preserved during code generation. This results in something that’s unpredictable. It results in the source prompts not telling you enough about what the software is doing and how it’s doing it, so you end up committing the generated code.

This also means that a comparison between compilers and LLMs is like comparing apples and oranges bicycles. Unless you prove me wrong.

A Tool, Not a Compiler

Just to clarify, I am not making an argument against AI. LLMs are useful tools, and I am not claiming otherwise.

What I am claiming is that LLMs are not compilers, and you can’t use them like you would compilers. I am also claiming that you can’t blindly trust the code they generate just because you trust the code compilers generate.