QLoRA on ExLlamaV3 Quants - Train @ 2-8bpw

EXL3-QLoRA
I've built this over the last few months and I think it's stable enough to share now. A fork of ExLlamaV3, this is a native ExLlamaV3 training implementation with plain PyTorch, without transformers.

exl3-qlora on github

Why?
With it, you can train 2-8bpw exl3 quantized models, rather than the old BNB 4 bit lock-in. EXL3 models are currently the highest precision models for their size, which leads to lower loss. Adapters are saved in standard PEFT format and load in TabbyAPI, PEFT, and can be merged back to the base model.

Model Support
Llama3, Mistral Small/Medium, Qwen3 dense and MoE, Qwen3.5/3.6/3.8 hybrid models, Gemma3/4 including MoE, Arcee Trinity. Each family is checked against the inference forward before training. Want more? I'm open to requests.

Features:

• SFT, DPO, KTO
• rsLoRA, PiSSA
• Optional LoRA on routed MoE experts, embedding, and LM head
• Single GPU, multi-GPU layer split, and DDP
• Sample packing, chat rendering through jinja, including tool calls and reasoning
• Gradient checkpointing, activation offload, chunked cross entropy, 8-bit and paged optimizers, Liger kernels
• Live run monitor in the browser (wandb at home)
• Real-time training, where the loaded model serves chat and trains its adapter in place

Benchmarks:

Qwen3.5-4B

Base	Eval loss	Peak VRAM
EXL3 6.0 bpw	2.128	12.7 GiB
EXL3 4.0 bpw	2.152	11.9 GiB
EXL3 3.0 bpw	2.165	11.5 GiB
bnb NF4 (axolotl)	2.183	12.8 GiB

All on the same dataset and hardware. bnb is about twice as fast, but exl3 is more flexible, vram efficient, and produces lower loss in my tests.

This is an unofficial community project, and a work in progress. Any feedback or feature requests are welcome, you can post them here, the github repo, or join us in the ExLlama Discord, where we have a thread in the #community-projects channel.

I have already produced several successful models with this training pipeline and released them, you can test them here.