Stanford teaches LLMs by making you build one

Stanford’s CS336 asks students to implement the tokenizer, model architecture, and optimizer needed to train a standard Transformer language model before they get to the harder parts: distributed training, FlashAttention2 in Triton, Common Crawl data cleaning, scaling laws, and alignment.

That is the point of the course. CS336 is built around the idea that language models are easier to understand if students construct the stack themselves. The course page compares the approach to operating systems classes where students build an OS from scratch. Here, the object is a language model pipeline: data collection and cleaning for pretraining, Transformer construction, model training, evaluation, and work before deployment.

The class is explicit about its expectations. Students need Python fluency, PyTorch familiarity, basic systems concepts such as memory hierarchy, calculus, linear algebra, probability, statistics, and prior machine learning background. It is a 5-unit class, and the warning is unusually direct: students get minimal scaffolding, and the amount of code is “at least an order of magnitude greater” than in other AI classes.

For engineers, that phrasing matters. A lot of current LLM usage is API-shaped. That is fine for many products, but it creates a thin understanding of what changes when a model gets slower, cheaper, less stable, easier to fine-tune, or more exposed to bad data. CS336 is aimed at the layer below that interface.

The assignments trace the parts of the system that usually disappear behind vendor documentation. Assignment 1 starts with basics: implement the core components and train a minimal language model. Assignment 2 moves into systems work: profile and benchmark the model and layers, optimize attention with a Triton implementation of FlashAttention2, and build memory-efficient distributed training code. Assignment 3 covers scaling by querying a training API to fit a scaling law. Assignment 4 turns raw Common Crawl dumps into usable pretraining data, including filtering and deduplication. Assignment 5 applies supervised finetuning and reinforcement learning to train language models to reason on math problems, with an optional section on safety alignment methods such as DPO.

That sequence is a useful curriculum because it refuses to separate model behavior from the machinery that produces it. Tokenization is not just preprocessing. Attention is not just a diagram in a paper. Deduplication is not just hygiene. Distributed training is not just an infrastructure bill. Each of those choices changes cost, throughput, failure modes, and what kind of debugging evidence you can trust.

The data assignment is especially relevant to security and infrastructure teams. Converting raw Common Crawl dumps into pretraining data sounds like plumbing until you remember that plumbing defines the model’s diet. Filtering and deduplication are not cosmetic steps; they are part of how teams control quality before training. Once a model has absorbed a dataset, the operational cost of finding and correcting bad inputs changes sharply.

The systems assignment is the other half of the lesson. Profiling, benchmarking, GPU efficiency, memory efficiency, and distributed execution are not optional details when the system is expensive to run. A model that works on one GPU in a notebook may fail the moment it is moved into a larger training job. Engineers responsible for these systems need to understand where memory goes, where kernels spend time, and how communication changes the shape of the problem.

The course page also includes practical compute guidance for people following along outside Stanford. It lists public pricing for a single B200 GPU on March 28, 2026: Modal at $6.25 per hour, Lambda Labs at $6.69 per hour, RunPod at $4.99 per hour, Nebius at $5.50 per hour or $3.05 per hour preemptible, and Together at $7.49 per hour with a minimum of 8 GPUs. The recommendation is to debug correctness on CPU first, then use GPUs for training runs or benchmarking GPU operations.

That advice is mundane and correct. Most wasted accelerator spend starts before the expensive run begins. If the tokenizer, shapes, loss calculation, checkpointing, or data pipeline are wrong, scaling the job mostly scales the mistake. CPU debugging is slow, but it is cheap evidence.

The honor-code section is also revealing. Prompting LLMs such as ChatGPT is allowed for low-level programming questions or high-level conceptual questions, but using them directly to solve the problem is prohibited. The course strongly encourages disabling AI autocomplete such as Cursor Tab or GitHub Copilot while completing assignments, saying that AI autocomplete makes it harder to engage deeply with the content.

That policy fits the course design. If the educational goal is to understand the full construction of a language model, outsourcing the construction defeats the exercise. For working engineers, the parallel is straightforward: generated code can be useful, but it can also hide the exact systems knowledge needed when a training run fails, a benchmark regresses, or an alignment method behaves differently than expected.

CS336’s premise is narrow and useful: to understand language models, build one end to end. The course does not treat LLMs as a single abstraction. It treats them as software, data, math, GPU kernels, distributed systems, and evaluation glued together into one expensive artifact. That is the part worth copying outside a university course.