Python Tooling at Scale: LlamaIndex’s Monorepo Overhaul
a year ago
- #Python
- #Monorepo
- #Tooling
- LlamaIndex manages a monorepo with over 650 Python packages, primarily Integrations and Packs.
- Each package has its own pyproject.toml file, defining dependencies, versioning, and tool configurations.
- The core package, llama-index-core, is a dependency for most other packages in the monorepo.
- Initial tooling included Poetry for individual package management and Pants for build orchestration.
- Challenges included slow dependency resolution, complex CI debugging, and high maintenance costs.
- The team migrated to uv (a faster Python project manager) and developed LlamaDev for monorepo-specific tasks.
- LlamaDev improved test speeds, logging clarity, and local reproducibility, reducing CI feedback loops.
- Contributors can now easily run LlamaDev locally, enhancing the development experience.