Aurora-80K releases! A modern tiny language model.

I'm introducing Aurora-80K, a small language model with exactly 80 thousand parameters.

It uses a factorized 4,096-token vocabulary despite having only 80K parameters.

The benchmarks:

Wikitext-2 BPB: 3.2902

BLiMP: 52.31%

Arc-Easy: 26.05%

More information about the model is available on the model page on Huggingface.

if there's any questions I'll happily answer them!