LoRA

Low Rank Adaptors (also known as LoRA) are a method to efficiently adapt large pretrained neural network models to downstream tasks while keeping the original model weights frozen. The key idea behind LoRA is to separately store updates to the pretrained weights as much smaller matrices, which are therefore much quicker to train and less prone to failure due to over training.

The LoRA technique was introduced in the paper LoRA: Low-Rank Adaptation of Large Language Models.

Technical Description

The key idea behind LoRA is to represent the update to the pretrained weights as low rank matrices that are injected into the original model. For a given layer with original weight matrix $W$ in $\mathbb {R} ^{d\times k}$ (i.e. a set of $d\times k$ real numbers typically represented as 16-bit floating point numbers), the low rank update is represented as:

$\Delta W=BA$

where $B$ in $\mathbb {R} ^{d\times r}$ and $A$ in $\mathbb {R} ^{r\times k}$ with rank r much less than the smaller of d and k (i.e. $r<<min(d,k)$). In this way a large matrix is represented as the product of two much smaller matrices, greatly reducing the amount of data involved.

During training, the original weight matrix $W$ is frozen and only $B$ and $A$ are trained, which dramatically reduces the number of trainable parameters. The forward pass is computed as:

$Y=XW+\alpha XAB$

where $X$ is the input to the layer. The scalar $\alpha$ controls the magnitude of the low rank update.

This allows adapting a huge pretrained model using only 0.01% additional trainable parameters stored in B and A. Since W remains fixed, LoRA does not add any inference latency or memory overhead.

Benefits and Performance

The key benefits of LoRA are:

• Massive reduction in trainable parameters during finetuning
• No increase in inference time or model size
• Ability to efficiently switch between tasks by swapping adapter weights
• Less GPU memory needed for training compared to full finetuning

LoRA has been shown to match the performance of full finetuning on models like GPT-2, while using several orders of magnitude fewer parameters. The QLORA paper combines LoRA with model quantization for additional efficiency gains. Overall, LoRA enables efficient deployment of gigantic pretrained models.

See Also

• LoRA: Low-Rank Adaptation of Large Language Models
• QLoRA