How Large Language Models Actually Work

The big idea in one sentence

A large language model (LLM) is a program that has learned, from enormous amounts of text, to predict the next piece of text given everything that came before it — and it does this one small piece at a time, over and over, until a full answer emerges.

That's it. Everything impressive an LLM does — writing code, summarizing a contract, explaining physics — comes from doing that one prediction extremely well. The rest of this article explains how.

Step 1: Text becomes tokens

Computers can't read words — they work with numbers. So the first thing that happens to your prompt is tokenization: the text is split into small chunks called tokens, and each token is mapped to an ID number.

Tokens are not exactly words. Common words are usually a single token, while rare or long words get split into pieces:

"Understanding transformers is fun!"

→ ["Under", "standing", " transform", "ers", " is", " fun", "!"]
→ [16648, 5646, 5276, 388, 374, 2523, 0]   (IDs are illustrative)

A tokenizer has a fixed vocabulary — typically tens of thousands to a few hundred thousand tokens — learned from data so that frequent character sequences get their own token. As a rough rule of thumb for English, one token ≈ ¾ of a word, so 1,000 tokens is around 750 words.

Why this matters to you: model pricing, speed and limits are all measured in tokens, not words. It also explains some odd failures — a model that "sees" strawberry as two or three chunks can struggle to count its letters.

Step 2: Tokens become vectors (embeddings)

A token ID like 2523 is just a label; it carries no meaning. So each token ID is looked up in a giant table and converted into an embedding: a list of hundreds or thousands of numbers (a vector).

These numbers are learned during training, and they end up encoding meaning. Tokens used in similar ways end up with similar vectors. In this "meaning space":

• cat sits close to dog and far from carburetor.
• Directions in the space can capture relationships — the classic example being that king − man + woman lands near queen.

The model also adds information about each token's position, because "dog bites man" and "man bites dog" contain the same tokens but mean very different things.

Step 3: Attention — tokens look at each other

Here's the breakthrough behind modern AI. In 2017, researchers at Google published "Attention Is All You Need", introducing the transformer architecture. Almost every major LLM today is a transformer.

The key mechanism is self-attention. A single token's embedding is ambiguous on its own — "bank" could be a river bank or a financial bank. Attention lets every token gather information from the other tokens around it to refine its meaning.

Consider: "The animal didn't cross the street because it was too tired." To understand it, the model needs to figure out that it refers to animal, not street. Attention is how it does that.

Mechanically, each token produces three vectors:

• Query — "what am I looking for?"
• Key — "what do I contain?"
• Value — "what information do I pass along if selected?"

Each token compares its query to every other token's key. High matches get high attention weights, and the token blends in those tokens' values. After this step, the vector for it contains information pulled from animal.

A transformer runs many attention "heads" in parallel (each can learn to track a different kind of relationship — grammar, references, topic) and stacks dozens of these layers, each followed by a small neural network. As the vectors flow up through the layers, they become richer and more abstract representations of the whole context.

Step 4: Predicting the next token

After the final layer, the model takes the vector at the last position and converts it into a score for every token in the vocabulary. Those scores are turned into probabilities:

Prompt: "The capital of France is"

" Paris"     91.2%
" a"          3.1%
" the"        1.7%
" located"    0.9%
...

One token is then chosen, appended to the text, and the entire process repeats to produce the next one. This loop — predict, append, repeat — is called autoregressive generation, and it's why answers stream onto your screen word by word.

How the token gets chosen is controlled by settings you may have seen:

• Temperature — low values (near 0) make the model pick the most likely tokens, giving focused, consistent output. Higher values flatten the probabilities, giving more varied and creative output.
• Top-p / top-k — restrict the choice to only the most likely candidates, cutting off unlikely tokens that might derail the text.

Generation stops when the model produces a special end-of-sequence token or hits a length limit.

How a model learns all this

None of the meaning in embeddings or the patterns in attention is programmed by hand. It's all learned, typically in three stages:

1. Pretraining

The model reads a massive dataset — trillions of tokens of web pages, books, code and more. The task is simple: hide the next token, have the model predict it, measure how wrong it was, and nudge all the parameters slightly to do better (this nudging is called gradient descent). Repeat this an astronomical number of times on large clusters of GPUs.

To get good at predicting the next token across all human text, the model is forced to absorb grammar, facts, reasoning patterns, coding conventions and much more. The result is a base model: knowledgeable, but it just continues text — ask it a question and it might respond with more questions.

2. Instruction tuning (supervised fine-tuning)

The base model is then trained further on curated examples of instructions paired with high-quality responses. This teaches it to behave like an assistant: answer the question, follow the format, stay on task.

3. Preference training

Finally, the model is refined using feedback about which responses are better. Techniques such as RLHF (reinforcement learning from human feedback) and related methods train the model to prefer answers that are more helpful, honest and safe. This stage shapes much of a model's "personality".

The context window

An LLM has no memory between conversations. Everything it knows about your chat is in the context window: the maximum number of tokens it can consider at once — your system instructions, the conversation so far, any documents you pasted, and the answer it's writing.

Context windows have grown from a few thousand tokens to hundreds of thousands or more. But "fits in the window" doesn't guarantee "used well": models can still overlook details buried in very long inputs, and longer contexts cost more and run slower. This is one reason techniques like retrieval-augmented generation (RAG) exist — to put only the most relevant information into the window.

Why models hallucinate

Remember the core job: produce the most plausible next token. Plausible is not the same as true. When a model lacks the right information, it can still generate fluent, confident text that sounds correct — a fake citation, a non-existent API function, a wrong date. This is called a hallucination.

Practical ways to reduce it:

• Ground it — provide the relevant source documents in the prompt and ask it to answer only from them.
• Allow uncertainty — explicitly tell it that "I don't know" is an acceptable answer.
• Ask for sources — request quotes or citations from the provided material so you can verify.
• Use tools — let the model call a search engine, calculator or database instead of guessing.
• Verify — treat important outputs as drafts to check, not final truth.

Key takeaways

• LLMs generate text one token at a time by predicting what comes next.
• Text → tokens → embeddings (vectors that capture meaning).
• Self-attention lets each token use context from all the others; transformers stack many attention layers.
• Temperature and sampling settings control how predictable or creative the output is.
• Models learn through pretraining, then instruction tuning and preference training.
• The context window is the model's entire working memory.
• Hallucinations come from optimizing for plausibility — ground, verify and use tools to reduce them.