Bilingual

WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
a month ago
- WAL-RUS is a Rust-based tool for Postgres backup and WAL archival, designed for predictable memory-efficiency and compatibility with WAL-G.
- WAL-G's Go-based implementation leads to unpredictable memory usage due to garbage collection, making resource planning difficult.
- WAL-RUS reduces peak virtual memory by over 70% compared to WAL-G, maintaining stable resource usage and efficient throughput.
- It maintains persistent connections for continuous WAL archival, optimizing for streaming workloads and minimizing buffering.
- Full compatibility with WAL-G archives and configuration allows straightforward migration for existing deployments.
Count the Bytes, Not the FLOPs
14 days ago
- The article describes how tsbootstrap's batched design, which materialized a large intermediate tensor (160 MB for 10,000 observations and 2,000 replicates), caused memory-bound performance issues due to low arithmetic intensity, leading to slower performance than the incumbent library.
- A fused, streaming path was implemented using kernel fusion to avoid materializing the full replicate tensor, reducing memory traffic by keeping the working set in cache and only storing the final statistics (e.g., 16 kB instead of 160 MB).
- A second performance wall was addressed by eliminating O(B) seed object allocations through a counter-based random number generator (Philox-4x32-10), reducing setup time from 13.1 ms to 0.55 ms for small workloads and ensuring bitwise reproducibility.
- The redesign achieved speedups of 3.1x to 20x on sustained workloads compared to the incumbent, with the fused path winning all 16 benchmark cells, though the materializing path still remains slower.
- Key principles: never materialize arrays only consumed by reductions, fuse operations to minimize memory traffic, and derive state rather than carrying it, leading to significant performance gains through subtraction rather than addition of complexity.
<antirez>
7 hours ago
- Redis Streams, introduced in Redis 5, are often mistakenly seen only as a Kafka-like messaging solution, but they are a general-purpose data structure.
- Streams overcome limitations of CSV logs by providing efficient range queries, auto-generated time-based IDs, compact memory usage, and the ability to remove old entries.
- Internally, Streams use delta-compressed macro nodes linked by a radix tree, enabling fast seek, range queries, and capped streams.
- Entry IDs are monotonically incrementing and time-based, allowing easy range queries by time (e.g., XRANGE).
- Streams can replace traditional patterns like Sorted Set + Hash for modeling objects (e.g., tennis matches), achieving over 13x memory savings (16.8 MB vs 220 MB for 1 million entries).
- This memory efficiency makes it viable to store billions of items in reasonable RAM, enabling new use cases like time series and small object tracking.
- Streams are not limited to streaming; they can be used for any scenario requiring ordered, append-only data with unique IDs and low memory overhead.
<antirez>
7 hours ago
- Amplitude published an article on scaling analytics using the Set data type, explaining why they avoided Redis due to overhead.
- Building custom vertical solutions offers control and debugging benefits but requires ongoing development and can become problematic if original developers leave.
- Redis already uses memory-efficient encodings for small data types (e.g., intset for numeric sets) but lacks a similar representation for small string sets.
- Antirez proposes a new hash table design using linear arrays per bucket with a high load factor to improve memory efficiency and cache locality.
- The new design must handle incremental rehashing, scanning, and serialization while potentially replacing both small encodings and general hash tables.

memory efficiency

WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups

Count the Bytes, Not the FLOPs

<antirez>

<antirez>