arXiv preprint · Jan 2026PEFT · LLMs
GRIT: fine-tuning with fewer trainable parameters
arXiv preprint 2601.00231, January 2026. Fourth of nine authors (first author P. Saha). Work done at RAAPID.
46%fewer trainable parameters than LoRA, on average
Read summary
GRIT is a parameter-efficient fine-tuning method. It trains about 46% fewer parameters than LoRA on average (25 to 80% depending on the task), while matching or beating LoRA and QLoRA.
It works by using information about how sensitive the model's loss is to each direction of change, and spending the small update only where it helps most. The update's size also changes during training.
The paper tests Llama-3.2 3B and Llama-3.1 8B on Alpaca, Dolly-15k, BoolQ, QNLI and GSM8K. On GSM8K, the 8B model reached the best accuracy in the comparison (0.662).
His part: experiment design, the reproducible evaluation setup, benchmark comparisons and ablations.