I just built a mini Kimi-K3 from Scratch under 250$. Already beats GPT-2 (124M)!

I pre-trained a 1.02-billion-parameter on Kimi K3 replica trained on 5.00 billion decontaminated tokens for $250.

This model has 1.02 billion parameters, of which 145 million are active per token.

It is roughly one two-thousandth of K3 by total size. It saw 5,000,003,584 tokens, which is a rounding error against the corpora frontier models are trained on.

It has never been instruction-tuned, and it has only ever done one thing: predict the next token.

What it does have is K3's architecture:

- Kimi Delta Attention, Gated MLA, Attention Residuals

- LatentMoE with the same aux-loss-free balancer

- Same activation function with the same two constants

- K3's own 163,840-token tokenizer, unmodified.

I report a 33.4% HellaSwag which beats the GPT-2 124M score of 28%

Read the entire tutorial here: https://books.vizuara.ai/book/pretraining-a-mini-k3