Editing LoRA (section)

== Technical Description ==
The key idea behind LoRA is to represent the update to the pretrained weights as low rank matrices that are injected into the original model. For a given layer with original weight matrix <math>W</math> in <math>\mathbb{R}^{d \times k}</math> (i.e. a set of <math>d \times k</math> [[wikipedia:real number|real numbers]] typically represented as 16-bit floating point numbers), the low rank update is represented as:

<math>\Delta W = BA</math>

where <math>B</math> in <math>\mathbb{R}^{d \times r}</math> and <math>A</math> in <math>\mathbb{R}^{r \times k}</math> with rank r much less than the smaller of d and k (i.e. <math>r << min(d,k)</math>).  In this way a large matrix is represented as the product of two much smaller matrices, greatly reducing the amount of data involved.

During training, the original weight matrix <math>W</math> is frozen and only <math>B</math> and <math>A</math> are trained, which dramatically reduces the number of trainable parameters. The forward pass is computed as: 

<math>Y = XW + \alpha X A B</math>

where <math>X</math> is the input to the layer. The scalar <math>\alpha</math> controls the magnitude of the low rank update.

This allows adapting a huge pretrained model using only 0.01% additional trainable parameters stored in B and A. Since W remains fixed, LoRA does not add any inference latency or memory overhead.