Boring Iceberg Catalog – 1 JSON file. 0 Setup
a year ago
- #data-engineering
- #iceberg-catalog
- #open-source
- Julien, a freelance data engineer based in Geneva, shares weekly insights on data engineering.
- Introduces the 'Boring Catalog', a simple Iceberg catalog implementation using a JSON file and S3 conditional writes.
- The Boring Catalog is designed for easy experimentation with Iceberg's commit mechanism, without complex setup.
- Features include a CLI tool 'ice' with commands like 'init', 'commit', and 'log', inspired by Git workflows.
- Supports Python and Polars interfaces for interacting with the catalog.
- Uses S3 conditional writes for concurrency control instead of traditional database backends.
- Future plans include enhancing the CLI and possibly adding a REST interface for broader integration.
- Encourages feedback and contributions, with a mention of Boring Data's onboarding package for data stacks.