Migrating 6.5B Rows from MongoDB
2 days ago
- Acquired Gauges analytics platform and promised to migrate historical data without requiring customers to change tracking code or API integrations.
- Replaced the Ruby on Rails application with a Laravel-based system to handle ingest and API compatibility, avoiding long-term reliance on an unfamiliar language.
- Built a secure, read-only migration API server to interface with the legacy MongoDB database, keeping the main Fathom application isolated from MongoDB.
- Mapped Gauges sites to Fathom accounts, handling both new and pre-existing customers, and migrated historical breakdown data as monthly aggregates to reduce request volume.
- Overcame bottlenecks including underpowered migration server, local ClickHouse memory limits, and AWS Lambda timeouts by moving to production and simplifying the migration interface.
- Successfully cut over tracking endpoints and API, imported 6.5 billion rows of data, and retired the old infrastructure, resulting in significant cost savings.