Text-to-LoRA: Hypernetwork that generates task-specific LLM adapters (LoRAs)
a year ago
- #lora
- #machine-learning
- #text-to-lora
- Install `uv` for dependency management and follow the installation guide.
- Clone the `text-to-lora` repository and set up the environment using `uv`.
- Install specific dependencies, including a custom wheel for flash attention.
- Download trained T2L models using Hugging Face CLI.
- Run a web UI demo locally with Mistral-7B-Instruct-v0.2 and T2L.
- Generate LoRAs from task descriptions via CLI, supporting models like Llama-3.1-8B and Gemma-2-2b.
- Evaluate generated LoRAs using scripts, with options for async validation via `watcher.py`.
- Train T2L models and oracle adapters for tasks, requiring significant GPU resources.
- Reconstruction training for T2L involves warmup, learning rate adjustments, and specific configurations.
- Performance comparisons show T2L outperforming baselines across multiple models and tasks.
- Note on non-deterministic behavior in vLLM with LoRA and dataset caching issues.