AI Engineering ยท 17 min read

GenAI + Agentic AI series #1

GenAI Foundations - How LLMs actually work

Understanding the working of LLMs

On this page
  1. "It's just fancy autocomplete"
  2. The one job: predict the next token
  3. Inside the model: from text to prediction
  4. 1. Tokenizer: text to numbers
  5. 2. Embeddings: numbers to meaning
  6. 3. Transformer layers: where the thinking happens
  7. 4. Output layer: meaning to probabilities
  8. Attention: the idea that made it all work
  9. The context window: the model's working memory
  10. Picking the next token: sampling
  11. How it learns to predict
  12. Scale: why "large" matters
  13. Why prediction looks like reasoning
  14. From base model to assistant
  15. Inference under the hood
  16. Where it breaks (and why)
  17. What this means for builders
  18. TL;DR

"It's just fancy autocomplete"

You've heard it. Maybe you've said it. And technically, it's true: a large language model (LLM) does one thing, over and over. It predicts the next piece of text.

But calling it "just autocomplete" is like calling a database "just a file". Accurate, and it tells you almost nothing about why the thing works, why it fails, or how to build on top of it.

This is the first article in my GenAI + Agentic AI series. Before we get to RAG, agents, MCP and production systems, we need a solid mental model of the engine underneath. By the end of this one you'll know:

  • What an LLM actually does, mechanically
  • What happens inside the model: tokens, embeddings, attention, layers
  • How the context window and sampling shape every response
  • How it learns, and why scale matters so much
  • Why simple prediction ends up looking like reasoning
  • How a raw model becomes a helpful assistant
  • What happens when you hit "send" (inference, KV cache, cost)
  • Where it breaks, and what that means for you as a builder

No heavy math. Just the stuff you need to stop treating it like magic.

The one job: predict the next token

Strip away the chat UI and an LLM is a function. Text goes in, and a probability distribution over what comes next comes out.

Tokens, not words. Models don't read words or characters. They read tokens, which are chunks of text. A common word like "the" is usually one token. A rarer word like "tokenization" might split into "token" + "ization". As a rough rule of thumb, one token is about three quarters of an English word. (Article 2 is all about tokens, because they quietly control your cost, latency and context limits.)

A distribution, not an answer. Give the model "The capital of France is" and it doesn't "know" the answer. It assigns a probability to every token in its vocabulary. " Paris" gets a very high score. " a", " located" and " the" get small ones. Everything else gets close to zero.

A loop, not a single shot. To produce a full response, the model runs this in a loop:

  1. Take the text so far (your prompt plus anything generated)
  2. Compute the probability of every possible next token
  3. Pick one (more on how it picks in a moment)
  4. Append it to the text
  5. Repeat until it emits a stop token or hits a length limit

An LLM writes one token at a time, in a loop: the text so far goes into the LLM, which produces next-token probabilities; one token is picked and appended, and this repeats until a stop token or the length limit.

That's it. Every essay, code snippet and "thoughtful" reply you've seen from an LLM came out of this loop, one token at a time. This is also why responses stream: the model is literally writing as it goes, with no draft it can go back and fix.

Inside the model: from text to prediction

So what actually happens inside one turn of that loop? Your text passes through four stages, and every modern LLM (GPT, Claude, Gemini, Llama and friends) follows roughly this shape.

Every token passes through four stages on its way to a prediction: the tokenizer turns text into token IDs using a BPE vocabulary; embeddings turn IDs into vectors plus position info; a stack of N transformer blocks (attention + MLP) refines each vector; the output layer produces logits and a softmax over next-token probabilities.

1. Tokenizer: text to numbers

The tokenizer splits your text into tokens and maps each one to an integer ID from a fixed vocabulary. Vocabularies are typically tens of thousands to a couple hundred thousand tokens. "Hello world" might become something like [9906, 1917]. The model never sees letters, only these IDs.

Most tokenizers use a method like Byte Pair Encoding (BPE): start from bytes, then repeatedly merge the most frequent pairs into new tokens. Common words become single tokens, rare words get split into pieces, and nothing is ever "out of vocabulary".

2. Embeddings: numbers to meaning

Each token ID is looked up in an embedding table and turned into a vector: a long list of numbers (often a few thousand dimensions). These vectors are learned during training, so tokens with related meanings end up close to each other in that space. "king" sits near "queen", "Go" sits near "Rust".

The model also needs to know where each token sits, since "dog bites man" and "man bites dog" use the same tokens. So position information is mixed in (modern models usually use a technique called rotary position embeddings, or RoPE).

3. Transformer layers: where the thinking happens

The vectors then flow through a stack of identical transformer blocks, often dozens of them. Each block does two things:

  • Self-attention: every token looks at the other tokens before it and pulls in the information it needs. (Next section.)
  • Feed-forward network (MLP): each token's vector is processed on its own through a small neural network. Researchers think a lot of factual knowledge is stored in these layers.

Each block refines the vectors a little. Early layers tend to capture surface stuff like syntax. Deeper layers capture more abstract things like meaning, relationships and intent. By the last layer, the vector at the final position holds the model's "understanding" of what should come next.

4. Output layer: meaning to probabilities

Finally, that last vector is projected back onto the whole vocabulary, producing one raw score (a logit) per token. A softmax function turns those scores into probabilities that sum to 1. That's the distribution the loop samples from.

One detail worth knowing: the model only predicts at the last position during generation, but it computes vectors for every token in the context. That's a big part of why long prompts are expensive.

Attention: the idea that made it all work

Before transformers (introduced in the 2017 paper Attention Is All You Need), language models mostly read text one word at a time, left to right, squeezing everything into a single running memory. Long-range connections got lost.

Attention fixes that. At every layer, each token can look directly at every earlier token and decide how much each one matters to it.

Take: "The server rejected the request because it was malformed." What does "it" refer to? To build a good vector for "it", the model needs to pull information from "request", not "server". Attention lets it do exactly that.

Mechanically, each token produces three vectors:

  • Query: what am I looking for?
  • Key: what do I contain?
  • Value: what will I hand over if someone attends to me?

Each token compares its query against every earlier token's key. High match means high attention weight. It then takes a weighted mix of those tokens' values. If you've worked with databases, think of it as a fuzzy lookup: the query is matched against all keys at once, and instead of returning one row you get a blend of rows, weighted by relevance.

Two more details:

  • Multi-head attention. Each layer runs many attention "heads" in parallel. One head might track grammar, another which noun a pronoun points to, another code indentation. Their outputs are combined.
  • Causal masking. During generation, a token can only attend to tokens before it, never after. That's what makes the model a next-token predictor: it can't peek at the answer.

The catch: every token attends to every earlier token, so the work grows roughly with the square of the context length. Double the prompt and attention gets about four times as expensive. Keep that in mind for the next section.

The context window: the model's working memory

The context window is the maximum number of tokens the model can process at once: your system prompt, the conversation history, any documents you pasted, tool results, and the response it's generating. Early models had a few thousand tokens. Today's frontier models handle hundreds of thousands, some over a million.

The important thing to understand: the context window is the model's only memory.

  • An LLM is stateless. Between API calls, it remembers nothing.
  • A "conversation" works because your app re-sends the whole history every time.
  • When the history outgrows the window, something has to be dropped, truncated or summarized. That's why chatbots "forget" early parts of long chats.

Bigger isn't automatically better, either:

  • Cost and latency scale with the tokens you send, on every single call.
  • Quality can drop in very long contexts. Models are often better at using information near the start or end of the prompt than information buried in the middle.
  • Noise hurts. Irrelevant text competes for attention with the text that matters.

This is why "just stuff everything into the prompt" rarely works in production, and why deciding what goes into the context (now called context engineering) is one of the core skills of building with LLMs. We'll go deep on it in Part III.

Picking the next token: sampling

The model gives you probabilities. Something still has to choose one token. That choice is called decoding or sampling, and it's controlled by settings you've probably seen in every LLM API.

Say the model is completing "My favorite language is" and the top candidates are:

TokenProbability
Go0.40
Python0.30
Rust0.15
Java0.10
everything else0.05

Here's how each strategy treats that list:

  • Greedy: always take the top token ("Go"). Deterministic, but tends to get repetitive and bland over long outputs.
  • Temperature: reshapes the distribution before sampling. Low temperature (like 0.2) sharpens it, so "Go" wins almost every time. High temperature (like 1.2) flattens it, so "Rust" or "Java" show up more often. Temperature 0 is effectively greedy.
  • Top-k: only sample from the k most likely tokens. With k = 2, only "Go" or "Python" can be picked.
  • Top-p (nucleus): only sample from the smallest set of tokens whose probabilities add up to p. With p = 0.85, that's Go + Python + Rust (0.40 + 0.30 + 0.15). It adapts: when the model is confident the set is small, when it's unsure the set grows.

Rule of thumb: low temperature for extraction, classification and code; moderate for general chat; higher for brainstorming and creative writing. Article 5 goes deeper on decoding and why even temperature 0 isn't always perfectly deterministic.

How it learns to predict

Nobody programs an LLM with rules. It learns by playing one game, billions of times: guess the next token, then check.

During pretraining, the model reads an enormous amount of text: web pages, books, code, papers. For every position in that text, it guesses the next token. When it's wrong, a training algorithm (gradient descent) nudges its internal numbers so the right token becomes a bit more likely next time.

How wrong was the guess? That's measured by a loss function (cross-entropy). If the model gave the correct token a probability of 0.9, the loss is small. If it gave it 0.01, the loss is large. Training is just a very long, very expensive process of pushing that number down across trillions of examples.

The nice part: this needs no human labels. The text is the answer key, since the next token is always right there in the data. That's called self-supervised learning, and it's why models can train on basically the whole public internet.

Those internal numbers are the parameters (or weights). Modern models have billions of them, and they start out random. After enough rounds of guessing and correcting, they encode a compressed picture of the patterns in the training data:

  • Language: grammar, style, tone
  • Facts: "Paris" tends to follow "capital of France"
  • Structure: how code, JSON, proofs and arguments are laid out
  • Associations: which ideas tend to appear together

Two things to keep in mind as an engineer:

  • The weights are frozen after training. The model isn't learning from your chats while it answers. Anything it "knows" was baked in before its training cutoff.
  • It stores patterns, not records. There's no lookup table of facts inside. This is the root of both its flexibility and its biggest flaw, which we'll get to.

Scale: why "large" matters

The "large" in LLM is doing a lot of work. Three things get scaled together:

  • Parameters: the size of the model. GPT-3 (2020) had 175 billion. Open models today commonly range from around a billion to hundreds of billions.
  • Data: the number of training tokens. GPT-3 trained on roughly 300 billion tokens. Meta's Llama 3 models were trained on over 15 trillion.
  • Compute: the GPU time to train. Frontier training runs use thousands of GPUs for months.

Researchers found that loss improves smoothly and predictably as you scale all three together. These are the scaling laws, and they're the reason labs kept betting on bigger models. Some abilities, like following complex instructions or multi-step reasoning, only become reliable past a certain scale.

What model size means for you as an engineer:

  • Memory: each parameter is usually stored in 16 bits (2 bytes). A 7B model needs about 14 GB just for its weights, a 70B model about 140 GB. That's before the memory needed for the context.
  • Quantization: storing weights in 8 or 4 bits shrinks that by 2x to 4x, with some quality loss. This is how people run models on laptops.
  • Speed and cost: bigger models are slower and more expensive per token. Picking the smallest model that does your job well is one of the easiest wins in production.
  • Mixture of Experts (MoE): many recent models only activate a fraction of their parameters per token, routing each token to a few specialist sub-networks. You get the knowledge of a big model at closer to the compute cost of a small one.

Why prediction looks like reasoning

Here's the part the "fancy autocomplete" crowd misses. To predict text really well, you have to model whatever produced the text.

Think about what it takes to predict the next token in these:

  • A Python function halfway through: you need to track variables, types and what the function is supposed to return.
  • A murder mystery's last page, "The killer was...": you need to have followed the plot.
  • A physics derivation: you need the steps to actually follow from each other.

A phone keyboard's autocomplete only looks at the last few words, so it never needs any of this. An LLM is trained on trillions of tokens with a huge context, and the cheapest way to get the loss down turns out to be learning internal representations of syntax, logic, facts and even other people's intentions. Nobody hard-coded those abilities. They emerged because they help with prediction.

Two behaviors fall out of this:

  • In-context learning. Show the model three examples of a format in your prompt and it follows the pattern on the fourth, with no retraining. The prompt itself steers the prediction.
  • Step-by-step reasoning. Ask it to "think step by step" and accuracy on multi-step problems often goes up. Writing intermediate steps gives the model more tokens to compute with before it commits to an answer. Modern "reasoning" models are trained to do this by default.

So is it really reasoning? That's a debate for philosophers. For engineers, the practical answer is: it does something that works like reasoning often enough to be useful, and fails in ways real reasoning wouldn't often enough that you can't trust it blindly.

From base model to assistant

A freshly pretrained model (a base model) is a brilliant document-continuer, and a terrible assistant. Ask it "How do I reverse a list in Python?" and it might continue with three more similar questions, because that's what a forum page full of questions looks like.

Turning it into the assistant you chat with takes extra training stages:

  1. Instruction tuning (supervised fine-tuning). Train on curated examples of instructions paired with good responses. The model learns the shape of "user asks, assistant answers".
  2. Preference tuning (RLHF and friends). Show the model pairs of responses, have humans (or other models) judge which is better, and train it to prefer the better kind. This shapes helpfulness, tone, honesty and safety.

Newer reasoning models add one more stage: reinforcement learning on tasks with checkable answers, like math problems and code with tests. The model is rewarded when its final answer is correct, and it learns on its own that writing out longer chains of thinking before answering gets rewarded more often. That's why these models "think" before they respond, and why they spend more tokens (and time) doing it.

The key insight: this doesn't change what the model fundamentally is. It's still predicting the next token. Fine-tuning just shifts the probabilities so that helpful, well-formatted answers become the most likely continuation of a conversation.

That's also why your system prompt matters so much. A chat isn't magic. It's one long document with roles marked inside it (system, user, assistant), and the model predicts what the assistant says next given everything above.

Inference under the hood

Training happens once. Inference, running the model to answer requests, happens millions of times a day, so this is where most engineering effort (and cost) goes. A request runs in two phases:

  • Prefill. The model processes your entire prompt in one parallel pass. GPUs are great at this, so it's relatively fast per token. This phase decides your time to first token.
  • Decode. The model generates the response one token at a time, each step depending on the last. This is sequential and can't be parallelized within a single request. This phase decides your tokens per second.

The KV cache. Remember the keys and values from attention? Without a cache, generating token 500 would mean recomputing keys and values for all 499 tokens before it. Instead, the model stores them after computing them once and only computes the new token's. This makes generation dramatically faster, but the cache eats GPU memory, and it grows with context length and the number of concurrent users. A lot of serving optimization is really KV cache management.

This explains a few things you've probably noticed:

  • Output tokens cost more than input tokens on most APIs, because decode is sequential and less efficient than prefill.
  • Long outputs feel slow while long prompts mostly just delay the first token.
  • Prompt caching discounts exist because providers can reuse the KV cache for a prompt prefix you send repeatedly, like a long system prompt.
  • Serving engines like vLLM exist mainly to batch many users' requests together and manage KV cache memory efficiently.

We'll cover serving properly in Part IV. For now, the takeaway: every token you send and every token you ask for has a real cost in time, memory and money.

Where it breaks (and why)

Every major LLM failure traces back to the same root: the model optimizes for plausible text, not true text. Most of the time those overlap. When they don't, you get these:

FailureWhat you seeWhy it happensWhat helps
HallucinationConfident, fluent, wrong answers. Fake citations, made-up APIsA plausible continuation scores high even when no real fact backs itGround it with RAG, ask for sources, verify outputs
Stale knowledgeMisses recent events, new library versions, your internal docsWeights are frozen at the training cutoff; private data was never in trainingInject fresh data via retrieval or tools
Tokenization quirksMiscounts letters, stumbles on some arithmeticIt sees token chunks, not individual characters or digitsHand math and string work to code or tools
Context limitsForgets or ignores details in very long inputsFixed token window; attention spreads thin over long inputsSend only relevant chunks; put key info first or last
Non-determinismSame prompt, different answersSampling picks from a probability distributionLower temperature, structured outputs, evals
Prompt sensitivitySmall wording changes swing the outputDifferent text, different context, different probabilitiesVersion prompts and run regression evals

None of these are bugs that will get patched in the next version. Models keep getting better at all of them, but they are properties of how the system works. That's good news, actually: once you understand the cause, you can design around it.

What this means for builders

The most useful mental shift: treat an LLM like a powerful but unreliable service, not like a function you can trust. You wouldn't call a flaky external API without timeouts, retries and validation. Same idea here.

In practice, that looks like:

  • Ground it in real data. Don't rely on what the model memorized. Put the relevant facts into the context (this is what RAG does, coming in Part II).
  • Give it tools for what it's bad at. Math, lookups, current data and actions belong in tools the model can call. This is the core idea behind agents and MCP.
  • Validate the output. Ask for structured output (JSON with a schema), parse it, and reject or retry anything that doesn't fit.
  • Control the randomness. Use low temperature for extraction and classification, higher for creative tasks.
  • Measure, don't vibe-check. Build a small eval set of real inputs and expected outputs, and run it every time you change a prompt or model.
  • Keep a human in the loop for risky actions. Anything irreversible (sending emails, closing tickets, moving money) should be gated.

I work on an AI pipeline that processes support tickets with LLMs and MCP tools, and almost every production issue we hit maps back to one of the failures in the previous section. Knowing why the model misbehaves is what lets you fix it with engineering instead of endless prompt tweaking.

TL;DR

  • An LLM predicts the next token, in a loop, one token at a time.
  • Inside: tokenizer to IDs, embeddings to vectors, a stack of transformer layers (attention + MLP), then softmax over the vocabulary.
  • Attention lets every token pull information from every earlier token. Its cost grows roughly with the square of the context length.
  • The context window is the model's only memory. The model itself is stateless.
  • Sampling settings (temperature, top-p) decide how predictable or creative the output is.
  • It learns by guessing and correcting over trillions of tokens. Scaling parameters, data and compute together makes it better, predictably.
  • Predicting well forces it to model language, logic and the world, which is why it looks like reasoning.
  • Instruction tuning, RLHF and RL on checkable tasks turn a document-continuer into an assistant without changing its core job.
  • Inference has two phases, prefill and decode, and the KV cache is why output tokens cost more.
  • It optimizes for plausible, not true. Hallucinations, stale knowledge and inconsistency come from that.
  • As a builder: ground it, give it tools, validate its output, and measure everything.

So yes, it's autocomplete. Just autocomplete that had to learn a model of the world to get good at its job.

Next up, Article 2: Tokens, Tokenization and Context Windows. The invisible unit that decides your bill, your latency and why your model "forgot" the first half of your prompt. Follow along so you don't miss it.