OpenZL: A Novel Data Compression Framework
7 hours ago
- #data-processing
- #OpenZL
- #compression
- OpenZL delivers high compression ratios while maintaining high speed, outperforming generic compressors.
- It generates specialized compressors optimized for specific data formats, compatible with a universal decompressor.
- Designed for engineers handling large specialized datasets, such as AI workloads, requiring fast processing pipelines.
- Under active development with evolving API, compressed format, and codecs, but maintains decompression compatibility across releases.
- Used extensively in production at Meta, considered production-ready.
- Supports building with `make` and `cmake`, offering various build types and modes for optimization and debugging.
- Includes settings for VSCode integration and requires generation of `compile_commands.json` for C++ language server support.
- Recommends using `clang-cl` for Windows builds for full C11 support, with alternatives like MinGW-w64 or limited MSVC support.
- BSD licensed.