Qwen 4 architecture: What do we know?

My best bet is the embedding-offloaded linear where the 51b n-grams track semantics and context, like 3.8 with its loss of real world knowledge bolted back on.

Otherwise? Sparse full-attention with dense routing? 6b active route to 'heavy' layers when the n-gram gets stuck.

Scenario 3: multi-head latent attention (mla) hybrid (reverse engineering Deepseek) compressing KV on the fly with the embeddings 'decompressing' on demand