I Survived the Zombie Apocalypse
3 days ago
- #Data Engineering
- #Firebase
- #BigQuery
- Firebase data can be exported to BigQuery for enhanced data analysis.
- Bruin is an open-source data platform handling data ingestion, transformation, quality, and governance.
- Bruin allows customers to run workloads on their own infrastructure for regulatory or preference reasons.
- The platform uses a distributed execution topology with a control plane (Orchestrator) and data plane (Bruin Agent).
- Challenges included authentication, error handling, and task status synchronization in a multi-tenant environment.
- Zombie tasks were identified where tasks were picked up but never completed or logged.
- Investigation revealed issues with HTTP timeouts and incorrect Go context usage.
- Silent failures in task reporting were fixed by adding proper error logging.
- A missing database index caused timeouts in task status updates, which was resolved.
- Lessons learned include the importance of proper context usage, error handling, and database optimization.