\n
\n
![\"NVIDIA](\"https:\/\/image.blockchain.news:443\/features\/D8E08E86F8EDBDDCD68414CF49BDD8B1401B11A69515DFF98E6B2B03EE9CF9D7.jpg\")
\n
\n <\/a>
\n <\/figure>\n
NVIDIA has announced the development of a new fine-tuning method called DoRA (Weight-Decomposed Low-Rank Adaptation), which offers a high-performing alternative to the widely used Low-Rank Adaptation (LoRA). According to the NVIDIA Technical Blog<\/a>, DoRA enhances both the learning capacity and stability of LoRA without introducing any additional inference overhead.<\/p>\n DoRA has demonstrated significant performance improvements across various large language models (LLMs) and vision language models (VLMs). For instance, in common-sense reasoning tasks, DoRA outperformed LoRA with improvements such as +3.7 points on Llama 7B and +4.4 points on Llama 3 8B. Additionally, DoRA showed better results in multi-turn benchmarks, image\/video-text understanding, and visual instruction tuning tasks.<\/p>\n This innovative method has been accepted as an oral paper at ICML 2024, marking its credibility and potential impact in the field of machine learning.<\/p>\n DoRA operates by decomposing the pretrained weight into its magnitude and directional components, fine-tuning both. The method leverages LoRA for directional adaptation, ensuring efficient fine-tuning. After the training process, DoRA merges the fine-tuned components back into the pretrained weight, avoiding any additional latency during inference.<\/p>\n Visualizations of the magnitude and directional differences between DoRA and pretrained weights reveal that DoRA makes substantial directional adjustments with minimal changes in magnitude, closely resembling full fine-tuning (FT) learning patterns.<\/p>\n In various performance benchmarks, DoRA consistently outperforms LoRA. For example, in large language models, DoRA significantly enhances commonsense reasoning abilities and conversation\/instruction-following capabilities. In vision language models, DoRA shows superior results in image-text and video-text understanding, as well as visual instruction tuning tasks.<\/p>\n Comparative studies highlight that DoRA surpasses LoRA in commonsense reasoning benchmarks and multi-turn benchmarks. In tests, DoRA achieved higher average scores across various datasets, indicating its robust performance.<\/p>\n DoRA also excels in vision language models, outperforming LoRA in tasks like image-text understanding, video-text understanding, and visual instruction tuning. The method’s efficacy is evident in higher average scores across multiple benchmarks.<\/p>\n DoRA can be integrated into the QLoRA framework, enhancing the accuracy of low-bit pretrained models. Collaborative efforts with Answer.AI on the QDoRA project showed that QDoRA outperforms both FT and QLoRA on Llama 2 and Llama 3 models.<\/p>\n DoRA’s application extends to text-to-image personalization with DreamBooth, yielding significantly better results than LoRA in challenging datasets like 3D Icon and Lego sets.<\/p>\n DoRA is poised to become a default choice for fine-tuning AI models, compatible with LoRA and its variants. Its efficiency and effectiveness make it a valuable tool for adapting foundation models to various applications, including NVIDIA Metropolis, NVIDIA NeMo, NVIDIA NIM, and NVIDIA TensorRT.<\/p>\n For more detailed information, visit the NVIDIA Technical Blog<\/a>.<\/p>\nAdvantages of DoRA<\/h2>\n
Mechanics of DoRA<\/h2>\n
Performance Across Models<\/h2>\n
Large Language Models<\/h3>\n
Vision Language Models<\/h3>\n
Compression-Aware LLMs<\/h3>\n
Text-to-Image Generation<\/h3>\n
Implications and Future Applications<\/h2>\n