2 months ago
- fenic is a semantic DataFrame library using Polars as its core execution engine for AI/LLM pipelines over unstructured data.
- Standard Polars lacks needed text operations (chunking, prompt templating, jq, fuzzy matching, markdown/transcript parsing, rich type casts).
- Python UDFs are too slow, break composition, and have poor type handling for these operations.
- The solution was building nine Rust expression plugins via pyo3-polars, making them native Polars expressions.
- Plugins run in-engine over Arrow buffers, are vectorized, composable, and have declared output types.
- Output type declaration is critical and can be static, derived from input fields, or from keyword arguments.
- The elementwise flag enables parallelism and streaming; broadcasting is handled via Polars helpers.
- Zero-copy Arrow access is achieved through a custom ArrowToValue trait with recursion for nested types.
- Plugins return pl.Expr, allowing composition with native Polars operations in a single expression tree.
- The build uses maturin with a single Rust crate; plugins also export plain Python functions for validation.