I implemented a modern LLM in 700 lines of C

I’ve been working on a small project called gemma4.c.

The idea is pretty simple: you can download a modern language model, compile one 700-line C file, and have it generate text on an ordinary CPU. Then you can read that same file from top to bottom and understand exactly how the model generates each new token.

The model is Gemma 4 E2B, one of Google’s latest open models. The C runtime handles the tokenizer, transformer, KV cache, sampling, and CPU kernels itself. There’s no inference framework or external library doing the interesting parts underneath it.

I built it mostly because I wanted to understand LLM inference at the level where it stops being diagrams and equations and becomes actual code. Keeping everything in one file made that much easier. You can start at main(), follow a prompt all the way through the runtime, see every buffer that’s allocated, every mathematical operation that transforms the activations, and every step that eventually turns your input into new tokens.

I ended up spending a lot of time on the CPU side too. The runtime uses int8 weights and activations, OpenMP, AVX2, and AVX-512 VNNI where available. On my Ryzen 7 7700 it gets about 639 tok/s on a 512-token prefill and 25.9 tok/s during generation, making it faster than llama.cpp.

The repo stays small on purpose. It only supports this model and CPU inference, so there’s much less machinery to work through than in a general-purpose runtime.

https://github.com/ryanssenn/gemma4.c