Teaching an LLM a Niche Diagraming Language
8 days ago
- #LLM Training
- #Diagram Generation
- #Pintora
- Text-to-diagram is well-solved for popular languages like Mermaid or PlantUML, but less so for niche languages like Pintora.
- Project goal: Train an LLM to generate and edit Pintora diagrams, focusing on models smaller than 30B.
- Base model selection: Qwen2.5-Coder-7B was chosen due to its balance of size and coding capabilities.
- Training phases: Continued Pretraining (CPT) to learn Pintora syntax, followed by Instruction Finetune (IFT) for task-specific abilities.
- Data preparation: Initially manual, then AI-generated due to scarcity, cleaned via scripts to ensure correctness.
- Training challenges: VRAM limitations led to renting a 48GB A40 GPU, with optimizations like removing unnecessary target_modules.
- Evaluation: Achieved 86% accuracy in generating syntactically correct Pintora diagrams from randomized prompts.
- Future directions: Exploring RL for accuracy improvement and potentially training an LLM for the music programming language Strudel.