LoRA: Low-Rank Adaptation of Large Language Models

LoRA: Low-Rank Adaptation of Large Language Models

Published	17 June 2021 (2021-06-17)
Authors	Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
Institution	Microsoft
arXiv Abstract	[[1]]
arXiv PDF	[[2]]

The paper LORA: Low-Rank Adaptation of Large Language Models proposes a method called Low-Rank Adaptation (LoRA) for efficiently adapting large pre-trained language models to downstream tasks.

Introduction

Pre-training language models on large amounts of text data and then fine-tuning them on downstream tasks has become a popular paradigm in NLP. However, as the size of pre-trained models increases (e.g. GPT-3 with 175 billion parameters), full fine-tuning becomes prohibitively expensive.

LoRA allows efficient adaptation by keeping the weights of the pre-trained model frozen, and injecting small trainable low-rank matrices into each layer. This greatly reduces the number of trainable parameters compared to full fine-tuning.

Method

For a pretrained weight matrix $W_{0}$ in $\mathbb {R} ^{d\times k}$, LoRA represents the update $\Delta W$ as a low-rank decomposition $\Delta W=BA$, where $B$ is $\mathbb {R} ^{d\times r}$, $A$ is $\mathbb {R} ^{r\times k}$, and $r<<min(d,k)$ is the rank.

• $W_{0}$ is frozen during training, only $A$ and $B$ are trainable.
• Can recover full fine-tuning flexibility by setting $r$ to rank of $W_{0}$.
• No additional inference latency, since $W=W_{0}+BA$ can be precomputed.
• Reduces memory usage during training up to 3x due to not storing optimizer states of frozen weights.

LoRA is applied to the query, key, and value matrices in the Transformer self-attention module.

Experiments

LoRA was evaluated on RoBERTa, DeBERTa, GPT-2, and GPT-3 on tasks including GLUE, summarization, and text generation.

• Matches or exceeds the performance of full fine-tuning despite having orders of magnitude fewer parameters.
• Outperforms adapter layers and prefix tuning baselines.
• Reduces GPT-3 checkpoint size from 350GB to 35MB, a 10,000x reduction.

Analysis

Further analysis provides insights into the low-rank adaptation:

• Adapting query and value matrices is better than just query or just value.
• Very low rank (e.g. 1-4) is sufficient for good performance.
• Learned low-rank matrices amplify directions not emphasized during pre-training.
• Magnitude of amplification can be very large (e.g. 20x).