Llama-Factory: Unified, Efficient Fine-Tuning for 100 Open LLMs
14 hours ago
- #AI
- #LLM
- #Fine-Tuning
- LLaMA Factory provides a comprehensive framework for fine-tuning large language models (LLMs) with various features and optimizations.
- Supports a wide range of models including LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen, DeepSeek, Yi, Gemma, ChatGLM, Phi, and more.
- Offers multiple training approaches such as pre-training, supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, etc.
- Includes scalable resources like 16-bit full-tuning, freeze-tuning, LoRA, and QLoRA via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ.
- Advanced algorithms supported: GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ, PiSSA.
- Practical tricks like FlashAttention-2, Unsloth, Liger Kernel, RoPE scaling, NEFTune, and rsLoRA are available.
- Supports wide tasks including multi-turn dialogue, tool using, image understanding, visual grounding, video recognition, audio understanding, etc.
- Experiment monitors include LlamaBoard, TensorBoard, Wandb, MLflow, SwanLab, etc.
- Faster inference options: OpenAI-style API, Gradio UI, and CLI with vLLM worker or SGLang worker.
- Provides extensive documentation, Colab notebooks, local machine setup, PAI-DSW, Alaya NeW, and official courses for getting started.
- Includes a changelog with updates on supported models and features, such as Qwen3, GLM-4.1V, InternLM 3, and more.
- Offers datasets for pre-training, supervised fine-tuning, and preference learning, with options for custom datasets.
- System requirements and installation instructions are detailed for different platforms including Windows, Ascend NPU, and AMD ROCm.
- Projects using LLaMA Factory include StarWhisper, DISC-LawLLM, Sunsimiao, CareGPT, and more, showcasing its versatility.
- Licensed under Apache-2.0, with model-specific licenses required for using corresponding weights.