Hasty Briefsbeta

All tags

redis

40 stories total

Bilingual

<antirez>
9 hours ago
- HyperLogLog is a randomized algorithm that approximates the number of unique elements in a set using constant, small memory.
- It uses hashing to assign elements to registers and counts the longest run of leading zeros to estimate cardinality.
- Redis implements HyperLogLog with 12KB per key, 16384 registers, and a standard error of 0.81%.
- The API includes PFADD, PFCOUNT, and PFMERGE commands, all operating on string-encoded HLL values.
- Performance optimizations include unrolled loops, precomputed lookup tables, and caching of the last computed cardinality.
- Bias correction for small cardinalities uses polynomial regression based on a four-order polynomial, improving accuracy in the range 40k–72k.
- The implementation uses a 64-bit hash function and 6-bit counters, eliminating practical limits on cardinality up to 2^64.
<antirez>
9 hours ago
- Distributed locks require mutual exclusion, deadlock freedom, and fault tolerance.
- A single Redis instance with replication has a safety violation due to asynchronous replication, causing potential lock conflicts during failover.
- For a single instance, a safe locking method uses SET with NX, PX, and a unique random value, releasing via a Lua script to avoid removing others' locks.
- The distributed algorithm uses N independent Redis masters (e.g., 5), where a client acquires locks on a majority (N/2+1) within a validity time, releasing unused locks promptly.
- Safety relies on ensuring a majority of keys exist simultaneously; liveness depends on auto-release of keys, client cooperation, and retries with random delays.
- Performance can be maintained by avoiding fsync, accepting a small risk of safety violation during crash-recovery, or enabling fsync=always for full safety but lower performance.
- Crash recovery without persistence can still be safe if restarted servers are kept unavailable until all previous locks expire.
- Clients can extend locks via scripts if operations take too long, preventing premature release.
<antirez>
9 hours ago
- A company experienced data loss when a Redis master with disabled persistence was restarted, causing a split-brain and replication of an empty dataset to all slaves.
- Redis Sentinel does not protect against misconfiguration like disabling persistence; this is a byzantine failure.
- Current Sentinel slave selection uses replication offset, which resets on restart and does not verify the master's runid, leading to potential promotion of outdated slaves.
- Proposed improvement: tie replication offset to the master's runid to ensure slaves have continuity with the failed master.
- Better Sentinel documentation is needed, including a list of common misconfigurations to avoid, such as turning off persistence.
<antirez>
9 hours ago
- Stripe engineers published a detailed report on Redis issues, leading to a Hacker News discussion.
- Redis now supports diskless replication, reducing the need for persistence in master-slave setups.
- Antirez suggested using average latency alongside the 99th percentile to identify outliers like EC2 spikes.
- This sparked Twitter backlash, with critics misinterpreting the suggestion as promoting averages over tail latency.
- Despite the misunderstanding, antirez plans to shift focus from Twitter to blogging and the Redis mailing list.
- He emphasizes that his time is better spent on development rather than unproductive Twitter arguments.
<antirez>
9 hours ago
- Redis persistence uses fork(), which causes latency spikes on Xen-based virtual servers like some EC2 instances.
- Stripe's latency graphs show delays up to one second during fork events, affecting the 99th percentile.
- Latency spikes every 30 minutes impact very few page views (about 0.05%), unlike evenly distributed spikes.
- For quality of service, the distribution of latency spikes matters more than average latency metrics.
- Redis is optimizing for EC2 with specific documentation and safer master-slave replication with disabled persistence.
- A quick fix for EC2 with disabled persistence is to disable automatic restarts and use Sentinel for failover.
<antirez>
9 hours ago
- Latency spikes during BGSAVE in Redis were not caused by disk I/O or the fork() call itself (fork took 6ms, spike ~300ms).
- Stack traces from Redis software watchdog showed the process was blocked on MOV operations after fork, indicating lazy page table copying by Linux.
- Linux optimizes fork() by not copying page table entries (PTEs) immediately; copy-on-write faults later cause delays, especially with many writes.
- The issue is not EC2-specific but more noticeable on virtualized instances due to slower PTE copying.
- Testing with jemalloc vs libc malloc revealed jemalloc caused higher latency spikes (339ms vs 21ms on bare metal) due to transparent huge pages (THP).
- Transparent huge pages used by jemalloc cause large copy-on-write workloads when the parent process touches shared pages after fork.
- The solution is to disable transparent huge pages using: echo never > /sys/kernel/mm/transparent_hugepage/enabled
- Building Redis with MALLOC=libc can reduce latency but may increase memory fragmentation; disabling THP is preferable.
<antirez>
9 hours ago
- Redis speed is a selling point, but comparative benchmarks are often misleading due to differences in data models, durability, and testing conditions.
- Fair comparisons require identical protocols and client libraries; otherwise, results reflect network stack and kernel rather than database performance.
- A benchmark comparing Redis and AerospikeDB on large EC2 instances (244 GB RAM) is unrealistic; a smaller m3.medium instance shows Redis achieving 32k ops/sec without pipelining.
- Using pipelining (e.g., 32 operations) boosts Redis performance dramatically: 250k ops/sec on a single core, outperforming the benchmark's multi-core result.
- Benchmarking simple GET/SET operations is insufficient; Redis excels at complex operations like ZADD for leaderboards and HyperLogLog for cardinality estimation, achieving 500k ops/sec combined.
- Better performance comparisons should focus on specific use cases, realistic instance types, and the number of instances needed to handle equivalent traffic.
<antirez>
9 hours ago
- Redis is now six years old, a significant milestone for the author who typically switches projects quickly.
- The author maintained focus on Redis by engaging in numerous side projects, both related and unrelated to Redis.
- Side projects like Load81, Dump1090, and Disque provided a creative outlet and prevented burnout.
- Allocating 20% of time to side projects like Disque is sustainable and ensures long-term commitment to Redis.
- Side projects often give rise to major projects; Redis itself originated as a side project of LLOOGG.
- Redis continues to grow and improve, attracting more developers and users over time.
- The author plans to continue with side projects to maintain endurance and innovation.
<antirez>
9 hours ago
- Travel from San Francisco to Sicily was difficult but worth it for the Redis Conference 2015.
- The conference was excellently organized by RackSpace with sponsors RedisLabs, Heroku, and Hulu.
- High-quality talks and around 250 attendees made the event valuable, with free admission and top-notch amenities.
- Notable talks included Hulu and Heroku on use cases and operational challenges, plus Bill Andersen's benchmarking vision.
- A negative point was Matt Stancliff's talk that criticized the author's leadership and proposed replacing him, which the author felt was unfair and poorly timed.
- Despite the disagreement, the conference was overall a magical experience, reinforcing the community's strength and the author's commitment to Redis.
<antirez>
9 hours ago
- DotScale 2015 conference was interesting; work focused on Sentinel for connection sharing and scaling.
- Redis 3.2 will include a Geo hashing API from Ardb, refactored by Matt Stancliff, merged as a priority.
- Bloom filters will be added in 3.2, likely as a new type with API control for accuracy/space tradeoff and auto-depollution.
- Two memory-related pull requests from RedisLabs will be merged to improve memory usage.
- A new memory introspection command will provide memory usage details similar to the LATENCY command.
- Redis Cluster will get multi-DC support via static slave options to avoid promotion in minority partitions.
- New list operations like LMERGE and O(N) operations for small N will be added.
- AOF safety feature includes optional RDB preamble for faster rewrites and reloads.
- SPOP COUNT option is already implemented for Redis 3.2.
- Redis Cluster redis-trib rebalance command will automatically rehash keys for balanced memory usage.
- Some features planned for 3.2 were ported to 3.0, like ZADD with NX and XX options.
- Work begins Monday, with first RC expected by end of September; stable release depends on bug reports.
<antirez>
9 hours ago
- The author values open source for its transparency, user-focused decisions, thorough documentation, and simplicity.
- Pivotal (and previously VMware) sponsored Redis, allowing it to grow as a pure open source project without commercial pressure.
- The author chose not to start a Redis company to preserve the project's magic and spend time with family, but supported an economic ecosystem around Redis.
- Redis Labs built a strong company with core developers and commercial products, and the author is joining them to work on open source while they focus on managed services.
- The move is expected to increase Redis Labs' contributions to the Redis core, such as memory reduction and key eviction improvements.
- The author expresses gratitude to Pivotal and Redis Labs for supporting the open source community.
<antirez>
9 hours ago
- Amplitude published an article on scaling analytics using the Set data type, explaining why they avoided Redis due to overhead.
- Building custom vertical solutions offers control and debugging benefits but requires ongoing development and can become problematic if original developers leave.
- Redis already uses memory-efficient encodings for small data types (e.g., intset for numeric sets) but lacks a similar representation for small string sets.
- Antirez proposes a new hash table design using linear arrays per bucket with a high load factor to improve memory efficiency and cache locality.
- The new design must handle incremental rehashing, scanning, and serialization while potentially replacing both small encodings and general hash tables.
<antirez>
9 hours ago
- Redis traditionally uses a single-threaded model with background I/O threads; the blocking DEL command can freeze the server for seconds on large keys.
- An initial incremental freeing approach used a timer to free memory gradually, adapting to memory pressure, but reduced performance to about 65% of normal.
- The final solution moved to a threaded lazy free implementation, which required eliminating shared robj structures from data types and client output buffers.
- Key changes include using dynamic strings in output buffers and SDS strings in data types, leading to faster performance and better memory efficiency.
- A new UNLINK command provides non-blocking deletion; FLUSHALL/FLUSHDB also gain lazy options.
- Future plans include threaded I/O and key-level locking for slow operations, moving toward a share-nothing design.
- The lazy free branch is currently under active development and targeted for Redis 3.4.
<antirez>
9 hours ago
- The Redis mailing list with 6700 members is being moved to Reddit, though the list will still exist for announcements and critical info.
- The mailing list is not as vital as it should be, with repetitive questions and important topics often overlooked.
- Twitter shows that people have moved away from email, but it's unsuitable for in-depth discussions.
- Reddit is chosen because it has a voting system to highlight quality content, a sidebar for FAQs, and gamification features.
- The plan is to take over the /r/redis subreddit and modify it to accept only comment submissions, eventually making Reddit the primary discussion forum.
- This is an experiment; if it fails, they may return to the mailing list or try Discourse.
- Email is no longer scalable for community discussions due to the volume of messages.
<antirez>
9 hours ago
- Lua scripting is the most successful Redis feature, enabling programmable and fast scripts that are cached and executed using the same interpreter.
- Two key features requested by users are a proper Lua debugger and script effects replication, both implemented by the author.
- The debugger is a remote, interactive, client-server tool that uses fork() to avoid blocking the server and enables rollback of dataset changes.
- Key debugger features include breakpoints, stepping, variable inspection, backtrace, and custom Lua functions like redis.breakpoint() and redis.debug().
- Script effects replication allows replicating only the write commands resulting from a script, improving efficiency and enabling use of non-deterministic commands like TIME.
- The feature supports opt-in via redis.replicate_commands(), with advanced control using redis.set_repl() to exclude certain commands from replication.
- Both features will be available in Redis 3.2, scheduled for release in December 2015.
<antirez>
9 hours ago
- Visualized all Redis commits as rectangles, where height represents affected lines on a logarithmic scale, with gray labels for release tags.
- Commit volume remained stable over time, but the rate of patch releases decreased after ceasing backports to version 3.0.
- Major releases (2.6, 2.8, 3.0) were initially more frequent, but future releases aim for three per year under a time-driven schedule.
- Patch releases for a given major version follow a logarithmic curve, with fewer critical bugs as the release matures and attention shifts to newer versions.
- The author wishes GitHub provided more advanced analytics and visualizations of commit data, which holds valuable project insights.
<antirez>
9 hours ago
- The algorithm generates monotonically increasing IDs using a majority of nodes with a 'current' key.
- It requires a data store supporting set_if_less_than and fsync on writes.
- Safety: It's impossible for two clients to generate the same ID or for a new ID to be less than a past ID.
- Liveness issue: concurrent accesses cause retries and 'holes' in the sequence.
- Performance is slow due to retries, but can be improved by a mediating ID server.
- Fsync is necessary to prevent ID regression after node restarts.
- The IDs provide total ordering useful for merging events across distributed processes.
- The algorithm resembles Paxos first phase and Raft leader election.
<antirez>
9 hours ago
- Redis creator initially avoided modules due to concerns about API compatibility and system identity.
- Lua scripting was insufficient for extending Redis beyond its core capabilities.
- Direct exposure of internal APIs to modules leads to fragility and evolution problems.
- A new modules API was prototyped to ensure binary and API compatibility across versions.
- The API provides low-level access to Redis data structures, enabling native-speed commands.
- An example module implementing 'list splice' is shown, with automatic memory management.
- Modules will be stable in Redis 4.0 with backward compatibility guaranteed.
- A modules directory and community-driven ranking system are planned.
- The API is still malleable and open for feedback.
<antirez>
9 hours ago
- Redis evicts old data when memory limit is reached, using policies like LRU and LFU.
- LRU (Least Recently Used) evicts keys with the greatest idle time, approximated by random sampling of keys.
- LFU (Least Frequently Used) evicts keys with the least number of accesses, using a logarithmic counter and time-based decay.
- Improvements include a pool of eviction candidates to avoid trashing good data and cross-database eviction.
- The 24-bit metadata field stores either LRU clock or LFU counter and last decrement time.
- New keys start with a base LFU value to avoid immediate eviction.
<antirez>
9 hours ago
- Redis 4.0 Release Candidate 1 is now available with major new features.
- It includes a modules system for custom data types and non-blocking commands.
- Replication version 2 (PSYNC2) allows partial resynchronization after topology changes.
- Cache eviction improvements include LFU and more robust policy implementations.
- Non-blocking operations: UNLINK for key deletion and ASYNC options for FLUSHALL/FLUSHDB.
- Mixed RDB-AOF persistence format offers faster rewrite and loading.
- The new MEMORY command provides detailed memory usage diagnostics.
- Redis Cluster now supports NAT/Docker environments, but requires a cluster reboot.
- This is an unstable release; future release candidates will lead to a final stable version.

firstprev0next

About|Login

redis

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>

<antirez>