System Info

Please update the forward of DoRA so that during evaluation it doesn't wastefully recompute weight normalization during each forward pass when it doesn't need to. Hopefully switching between the current implementation and an implementation that precomputes the weight norm is as simple as checking model.eval() # not model.training.

I understand one could merge the weights but it tends to be more lossy than simple unmerged adapters. Additionally, this should speed up evals during finetuning larger models.

Who can help?

@BenjaminBossan

Reproduction

https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/dora.py#L66-L101

Expected behavior

Inference with unmerged adapters (model.training==False) does not compute weight norm during each forward pass.