LoRA.

LoRA (Low-Rank Adaptation) is a breakthrough fine-tuning method that enables efficient customization of large language models without modifying their original parameters. Instead of updating billions of weights during training, LoRA adds small trainable matrices that capture task-specific adaptations while keeping the base model frozen. This approach dramatically reduces computational costs and memory requirements, making specialized model variants accessible to researchers and developers with limited resources.

The technique works by decomposing weight updates into low-rank matrices, typically reducing trainable parameters by 99% or more compared to full fine-tuning. LoRA inserts trainable rank decomposition matrices into transformer layers, learning adaptation patterns that can be applied on top of the base model during inference. The key insight is that most fine-tuning changes have low intrinsic dimensionality, meaning they can be captured effectively with much smaller parameter sets than the full model would suggest.

LoRA's composability allows multiple adaptations to be stored, swapped, or even combined for different tasks without maintaining separate full model copies. This modularity enables rapid experimentation and deployment of specialized variants while preserving the base model's general capabilities. However, LoRA may not capture complex adaptations as effectively as full fine-tuning for dramatically different domains, and the rank selection requires careful tuning to balance efficiency with adaptation quality.