Generative AI L29: Low Rank Adaptation (LoRA)

This lecture, part of the ‘Foundations of Generative AI’ course at LUMS, focuses on Low Rank Adaptation (LoRA), a parameter-efficient fine-tuning technique. The instructor begins by reviewing previous topics and addressing student questions about adaptive methods and the role of non-linearity in neural networks. He then introduces the core premise of LoRA: weight changes during fine-tuning have low intrinsic dimensionality. The lecture explains how a large weight update matrix can be approximated by the product of two smaller matrices (A and B), leading to significant memory savings. Using simple examples, the instructor illustrates matrix factorization, rank, and the break-even point where the compressed representation becomes less efficient. He emphasizes that LoRA works well because the weight updates in fine-tuning typically lie in a low-dimensional subspace, and he discusses practical considerations such as choosing the rank hyperparameter and initialization strategies. The lecture concludes with a discussion on why B is initialized to zero and A to random values, ensuring stable training.