What CI looks like at a 100-person team (PostHog)
5 days ago
- #AI
- #FlakyTests
- #CI/CD
- PostHog's CI ran 575,894 jobs, processed 1.18 billion log lines, and executed 33 million tests in one week.
- Mendral is an AI agent designed to diagnose CI failures, quarantine flaky tests, and open PRs with fixes.
- PostHog's CI infrastructure handles 65 commits merged to main per day and 105 PRs tested per day.
- At scale, even a 99.98% pass rate results in significant noise due to the sheer volume of test executions.
- Mendral ingests logs at scale, detects and traces flakes, opens PRs with fixes, and acts as a team member on Slack.
- Flaky tests are rarely random; they often have deterministic root causes like timing dependencies or shared state.
- The routing problem—knowing who to notify about a failure—is as critical as diagnosing the failure itself.
- Working on a public repo like PostHog's ensures transparency and accountability for Mendral's actions.
- AI coding tools are increasing the volume of code changes, making CI challenges grow for most teams.
- PostHog's engineering team exemplifies a fast-moving, well-run operation with 22,477 tests and a 99.98% pass rate.