Memlz: Fast compression library for C/C++ on x64/x86
5 months ago
- #high-performance
- #library
- #compression
- memlz is a high-speed compression library designed for scenarios requiring speeds close to memcpy().
- Unlike other fast libraries (Snappy, FastLZ, LZAV, LZO), memlz achieves speeds comparable to LZ4 with specific acceleration settings.
- Benchmarks on Intel i7 show decompression speeds vary by data type, with non-cached memcpy() at 14000 MB/s.
- Header-only library; simple integration via `memlz_compress()` and `memlz_stream_compress()` for streaming mode.
- Streaming mode supports compressing data larger than memory or improving ratio for small packets.
- Includes functions to check compressed and decompressed sizes (`memlz_compressed_len()`, `memlz_decompressed_len()`).
- Guarantees safe decompression: no out-of-bounds memory access and regular completion time.
- Eliminates the need for an internal payload queue in streaming mode, unlike LZ4.
- Integration into eXdupe file archiver shows slight speed improvement in streaming mode without significant ratio loss.
- Early beta version; tested only on Intel machines; compatibility may be unstable.