# GRIT: Geometry-Aware PEFT with K-FAC Preconditioning, Fisher-Guided Reprojection, and Dynamic Rank Adaptation

**Venue:** arXiv preprint (cs.LG), not peer reviewed · **Date:** 2026-01-01

**Authors:** Pritish Saha, Chandrav Rajbangshi, Rudra Goyal, Mohit Goyal, Anurag Deo, Biswajit Roy, Ningthoujam Dhanachandra Singh, Raxit Goswami, Amitava Das

**My role:** Fourth of nine authors, at RAAPID Lab. Experiment design, reproducible evaluation setup, benchmark comparison and ablation analysis.

**Tags:** PEFT, LoRA, fine-tuning, K-FAC, Fisher information, LLM optimization, evaluation

---

## The finding

GRIT is a fine-tuning method for large language models. The authors report that it trains about 46% fewer parameters than LoRA on average. In Table 2 the reduction is 26.6 to 80% per row. On GSM8K with Llama-3.1 8B, GRIT scores 0.6619 accuracy against 0.6338 for LoRA, with 27.8% fewer trainable parameters. Across the ten rows of Table 2, GRIT is above LoRA on four and below on six. The largest shortfall is 0.0086, on QNLI with Llama-3.1 8B (0.9211 against 0.9297). GRIT is above QLoRA on nine of the ten rows. The paper is an arXiv preprint and has not been peer reviewed.

## The problem

Fine-tuning adapts a pretrained model to a task. Retraining every weight is costly, so parameter-efficient fine-tuning (PEFT) trains a small set of added weights instead. LoRA is the standard form. It writes the weight change as a product of two thin matrices, B and A, with a small inner size called the rank. QLoRA is LoRA on top of a quantized base model, one stored in 4-bit numbers to save memory.

The paper argues that these methods ignore the shape of the loss surface. They use fixed rank, randomly oriented low-rank directions and plain first-order steps. That can waste the update budget on directions the task does not need, and push the model along directions that the pretrained model depends on. The authors link this to drift and forgetting: the model gets better at the new task and worse at what it knew before.

## How GRIT works

GRIT keeps LoRA's form and changes how the adapter is trained. The method has three parts.

### 1. K-FAC preconditioning in rank space

Natural-gradient training rescales each step by the curvature of the loss, which is how sharply the loss bends. Steps along sharp directions get smaller. Full curvature matrices are too large for an LLM. K-FAC (Kronecker-factored approximate curvature) approximates them with two smaller matrices per layer, one for the layer's inputs and one for its output gradients. GRIT computes these in the rank space of the adapter, so each matrix is only r by r (r is the LoRA rank). The authors add a small damping value to the diagonals, a small constant that keeps the matrix inversion numerically safe. They use the inverses to rescale the gradients of A and B. Statistics are collected every few steps, and natural-gradient steps start after a warm-up.

### 2. Fisher-guided reprojection

The same two matrices approximate the Fisher information, which measures how strongly the loss reacts to a change along each direction. Every T steps, GRIT takes the eigenvectors of both matrices and keeps the top ones. Eigenvectors are the main directions of a matrix, and each has an eigenvalue that says how much it carries. GRIT projects A and B onto the kept directions. A blending factor mixes the old and projected values so the change is gradual.

The paper lists different reprojection settings in different tables. Tables 5 and 6 give T = 200 steps. The Table 10 configuration defaults list reprojection_freq = 50. The timing runs in Table 8 measured 1.8 to 2.4 reprojections per 1,000 steps.

### 3. Dynamic rank

Rank is the number of directions an adapter uses. GRIT sets it per module from the Fisher spectrum. It keeps the smallest number of directions whose eigenvalues cover an energy threshold, within a minimum of 4 and a maximum of 16 (3B model) or 32 (8B model). The method section and Table 5 give the threshold as 0.90. Table 10 lists 0.99. Dropped directions are only suppressed, not deleted, so they can return if their eigenvalues grow.

### A forgetting law

Earlier work models forgetting with a power law: loss on pretraining data after fine-tuning grows with the amount of fine-tuning data and shrinks as the model gets larger. Read concretely, more fine-tuning data makes the model forget more, and a bigger model forgets less. GRIT extends the law with an efficiency factor that multiplies model size. The factor grows with the effective rank, how well updates align with the Fisher directions, and how much projection is applied. In the paper's account, more alignment at the same budget means less forgetting. The paper says the scaling constants in the factor are fitted from runs that vary data size, rank and reprojection frequency.

### Setup

Models: Llama-3.2 3B and Llama-3.1 8B, with NF4 4-bit quantization (a 4-bit format that stores the frozen base weights in far less memory) and bf16 compute (a 16-bit number format). Datasets: Alpaca, Dolly-15k, BoolQ, QNLI and GSM8K. The optimizer is AdamW on the preconditioned gradients, with cosine decay, 5% warm-up and gradient clipping at 1.0. The paper lists seeds 41, 42 and 43. Baselines: LoRA, QLoRA, IA3, DoRA, orthogonal-LoRA and Shampoo. GRIT reuses the QLoRA data pipeline.

## Results

Source: Table 2 of the paper, which reports absolute scores and the percentage reduction in trainable parameters relative to LoRA. The metric here is ROUGE-1 for Alpaca and Dolly-15k, and accuracy for the other three tasks. Trainable parameters are in millions (M).

| Model | Task | LoRA | QLoRA | GRIT | Params, LoRA | Params, GRIT |
|---|---|---|---|---|---|---|
| Llama-3.2 3B | Alpaca | 0.1852 | 0.1292 | 0.1844 | 24.31M | 8.45M (65.3% fewer) |
| Llama-3.2 3B | Dolly-15k | 0.1733 | 0.1108 | 0.1976 | 24.31M | 17.01M (30% fewer) |
| Llama-3.2 3B | GSM8K | 0.3935 | 0.3836 | 0.3867 | 24.31M | 15.3M (37% fewer) |
| Llama-3.2 3B | QNLI | 0.8938 | 0.8885 | 0.9053 | 24.31M | 7.75M (68.1% fewer) |
| Llama-3.2 3B | BoolQ | 0.7834 | 0.7525 | 0.7749 | 24.31M | 15.03M (38.2% fewer) |
| Llama-3.1 8B | Alpaca | 0.2036 | 0.1402 | 0.2034 | 83.89M | 19.27M (77% fewer) |
| Llama-3.1 8B | Dolly-15k | 0.1921 | 0.1272 | 0.1968 | 83.89M | 54.3M (35.2% fewer) |
| Llama-3.1 8B | GSM8K | 0.6338 | 0.6315 | 0.6619 | 83.89M | 60.5M (27.8% fewer) |
| Llama-3.1 8B | QNLI | 0.9297 | 0.9248 | 0.9211 | 83.89M | 16.6M (80% fewer) |
| Llama-3.1 8B | BoolQ | 0.8345 | 0.8229 | 0.8336 | 83.89M | 61.56M (26.6% fewer) |

GRIT scores above LoRA on four of the ten rows (Dolly-15k on both models, QNLI on 3B, GSM8K on 8B) and below on six. The largest shortfall against LoRA is 0.0086, on QNLI with the 8B model (0.9211 against 0.9297). The next largest is 0.0085, on BoolQ with the 3B model (0.7749 against 0.7834). GRIT is above QLoRA on nine of ten rows. The exception is QNLI on 8B (0.9211 against 0.9248).

Every row uses fewer trainable parameters than LoRA, from 26.6% to 80% fewer. The 46% average is the paper's figure.

Cost, from Table 8 (an 8B model, three task setups): mean step time is 236, 232 and 242 ms for GRIT against 228, 224 and 235 ms for QLoRA, about 3 to 4% more. Peak memory rises by about 0.7 GB. The paper's text gives a wider range of 6 to 10% step overhead.

The ablations in Appendix A.11 are described in words only. Removing K-FAC gives small but consistent drops on instruction and classification tasks, with higher training variance. Removing reprojection raises effective rank and forgetting. A fixed rank keeps low-energy directions and saves fewer parameters. The default damping is 1e-3. The paper says larger values weaken the preconditioning and smaller ones can destabilize the inverses.

## Figures

### Figure 1(a)

![Scatter plot of model weights in a two-dimensional projection. A wide blue cloud shows the base weights. A small red cluster and a small green cluster at its center show the weight changes from LoRA and GRIT.](https://mohitx.in/research/grit/overall_new.webp)

*GRIT and LoRA both change the model inside a small region of weight space.*

### Figure 1(b)

![Zoomed scatter plot comparing LoRA weight changes in red with GRIT weight changes in green. The green points form a tighter, rounder core.](https://mohitx.in/research/grit/grit_parameters.webp)

*Zoomed in, GRIT's weight changes spread less than LoRA's. This is a qualitative picture, not a measurement of forgetting.*


## Limits

- The paper states that GRIT relies on approximate curvature from K-FAC. Mis-specified damping or noisy spectra could cause misalignment.
- Experiments cover two Llama backbones and five mostly short-context, English text benchmarks. The paper says multimodal, multi-turn agent and RLHF settings are untested.
- The paper lists sensitivity to the reprojection period and the energy threshold, and dependence on the FSDP or DeepSpeed setup.
- The lower-drift claim rests on pretraining-loss proxies. The paper says gold-standard drift measures are expensive and are approximated.
- Table 2 also lists Q-GRIT, a quantized variant. Its results against QLoRA are mixed: for example, 3B QNLI accuracy is 0.8449 against 0.8885.
- The main tables compare GRIT with other PEFT methods and have no full fine-tuning row.
- This is a preprint and has not been peer reviewed.

## My contribution

I am the fourth of nine authors. My part was experiment design, the reproducible evaluation setup, benchmark comparison and ablation analysis. Pritish Saha is first author. The method and the paper are the work of the full team at RAAPID Lab.

## Cite

```bibtex
@misc{saha2026grit,
  title         = {GRIT: Geometry-Aware PEFT with K-FAC Preconditioning, Fisher-Guided Reprojection, and Dynamic Rank Adaptation},
  author        = {Saha, Pritish and Rajbangshi, Chandrav and Goyal, Rudra and Goyal, Mohit and Deo, Anurag and Roy, Biswajit and Singh, Ningthoujam Dhanachandra and Goswami, Raxit and Das, Amitava},
  year          = {2026},
  eprint        = {2601.00231},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  doi           = {10.48550/arXiv.2601.00231}
}
```

## Links

- [arXiv abstract](https://arxiv.org/abs/2601.00231)
- [arXiv HTML version](https://arxiv.org/html/2601.00231)
- [DOI](https://doi.org/10.48550/arXiv.2601.00231)