We Saved $500k per Year by Rolling Our Own S3

6 months ago

Nanit built N3, a Rust-based in-memory landing zone, to replace S3 for their video processing pipeline, saving $500,000 annually.
S3's PutObject request fees and 1-day minimum storage lifecycle were the main cost drivers, as videos were processed in ~2 seconds but stored for 24 hours.
N3 holds videos in memory for ~2 seconds, using S3 only as an overflow buffer, maintaining reliability and strict per-baby ordering.
The architecture includes N3-Proxy (stateless, handles uploads) and N3-Storage (stateful, holds videos in RAM), with fallback to S3 when N3 is overloaded.
Key optimizations included DNS-based load balancing, network-optimized instances, Rust-native TLS (rustls), and disabling TCP timestamps to reduce ACK overhead.
Memory leaks were addressed by enforcing connection timeouts and disabling keep-alive, while graceful restarts ensured no data loss during pod updates.
The system uses delete-on-download and TTL garbage collection to manage memory, with S3 as a safety net during maintenance or processing delays.
Early validation via synthetic stress tests and a production PoC (mirror mode) de-risked the design, revealing bottlenecks like TLS overhead and network throttling.
The project succeeded due to clear problem definition, early validation, and optimizing for the happy path while relying on S3 for edge cases.
Custom infrastructure like N3 is justified when scale and specific constraints enable a simpler, cheaper solution than managed services.

Hasty Briefsbeta