Dataframely: A polars-native data frame validation library
a year ago
- #polars
- #python
- #data-validation
- QuantCo migrated data pipelines from pandas to polars for performance gains.
- Legacy codebases lacked clarity in data frame invariants, leading to inefficiencies.
- Existing validation libraries like pandera and patito had limitations for polars.
- dataframely was developed to address these shortcomings with polars-native support.
- Features include schema definition, validation, interdependent data frame checks, and test data generation.
- Soft-validation and failure introspection improve debugging in production pipelines.
- dataframely enhances code readability, robustness, and static type checking.
- Used successfully in multiple teams for analytical and production pipelines.
- Open-sourced to benefit the broader data engineering community.