Bilingual

Client-side load balancing at a million requests per second
25 days ago
- Client-side load balancing (CSLB) was built to replace shared edge ingress load balancer for internal fan-out traffic, handling over a million requests per second.
- Key features added include N-ring fade-in for gradual scale-up, occupancy-based bounded load for even distribution, and AZ-aware routing with latency health factors.
- Migration from Skipper to CSLB reduced latency spikes, improved stability, and cut costs by scaling down Skipper fleet and reducing pod count through better load management.
- Implementation involved hash parity with Skipper using xxHash64, Kubernetes discovery via watch-based informer, and a hardened pipeline for faster, safer deployments.
- Hardening measures included retry policies, FIFO buffers, and enhanced logging, which revealed and mitigated node-level network freezes without causing incidents.
- Lessons learned: owning routing decisions provides valuable telemetry but adds complexity; cache locality conflicts with traffic isolation; and fast deployment pipelines reduce risk.
- Future work includes resuming AZ-aware routing trials to evaluate cost savings versus cache fragmentation, especially during peak traffic events.
Rewriting Bun in Rust
17 days ago
- Bun started as a Zig port of esbuild's JavaScript/TypeScript transpiler and grew into a large-scope project including a package manager, test runner, and Node.js APIs.
- Stability issues like memory leaks and crashes in Zig led the team to rewrite Bun in Rust using Claude AI, completing the port in 11 days with extensive automated workflows and adversarial reviews.
- The Rust rewrite fixed 128 bugs, reduced memory usage, decreased binary size by ~20%, improved performance by 2-5%, and enhanced maintainability with Rust's safety features.
- Key challenges included mapping Zig patterns to Rust, handling compiler errors, and ensuring zero regressions through a comprehensive test suite and adversarial code review processes.
- The project used 64 Claude AI agents at peak, processed billions of tokens, and resulted in 6,778 commits, demonstrating the scale of AI-assisted code rewriting.
- Future improvements will leverage Rust's borrow checker, Miri, LeakSanitizer, and continuous fuzzing to further enhance stability and performance.
Data-Oriented Design (2009)
12 days ago
- Data-oriented design is proposed as an alternative to object-oriented programming (OOP) for game development.
- OOP often leads to performance issues like cache misses and difficult parallelization.
- Data-oriented design focuses on data layout, aiming for contiguous, homogeneous blocks for efficient processing.
- It improves cache utilization, simplifies parallelization, and enhances modularity and testing.
- Drawbacks include unfamiliarity and integration challenges with existing OOP code.
- OOP may still be suitable for GUI systems or when dealing with singular entities.
Static search trees: 40x faster than binary search (2024)
8 days ago
- The post implements an optimized static search tree (S+ tree) for high-throughput searching of sorted 32-bit unsigned integers, focusing on throughput measured as queries per second.
- Key optimizations include using SIMD instructions for node searches, batching queries to parallelize memory accesses, prefetching to reduce latency, and pointer arithmetic to streamline operations.
- Tree layouts like S+ trees with branching factor 16 or 17 are explored, with interleaved queries balancing CPU and memory-bound workloads, achieving over 40x speedup from 1150 ns/query to 27 ns/query for 4GB data.
- Prefix partitioning by top bits is tested but adds complexity with limited speed gains, especially on skewed real data like human genome k-mers, making it less practical than interleaved queries.
- Future work suggestions include branchy search for variable iterations, interpolation search to reduce RAM accesses, packing data into 16 bits, supporting range queries, sorting queries for cache reuse, and applying to suffix array searching.

Hasty Briefsbeta

performance-optimization

Client-side load balancing at a million requests per second

Rewriting Bun in Rust

Data-Oriented Design (2009)

Static search trees: 40x faster than binary search (2024)