A Math-Light Walkthrough of How Transformer LLMs Actually Work

This explainer breaks down the inner workings of transformer-based LLMs without diving into the underlying math, aimed at readers who want to understand model cards and papers without needing a research background. It walks through the full pipeline: tokenization (turning text into integer IDs via subword vocabularies like BPE or SentencePiece), embeddings (mapping those IDs to learned vectors where semantically similar tokens cluster together), and positional encoding (injecting word-order information that raw self-attention lacks).

The piece highlights why modern architectures have largely moved from the original sinusoidal additive positional encodings to Rotary Position Embeddings (RoPE), now standard in LLaMA, Mistral, Gemma, and Qwen. RoPE rotates token vectors by position-dependent angles rather than adding position vectors, naturally encoding relative distance, generalizing better to longer contexts, and adding no extra parameters. It also touches on quirks like the ‘how many R’s in strawberry’ problem, which stems from models operating on token IDs rather than individual characters.

The broader point is that most modern LLMs share the same transformer skeleton, and the real differences come from training data, scale, configuration, and post-training. Understanding the components — attention, multi-head attention, feed-forward networks, residual streams, and the generation loop — gives readers the vocabulary to parse architectural choices across the current open-weight model landscape.