Telstra outage: The night a network decided the year was 2006
8 hours ago
- A GPS receiver in Melbourne, after scheduled maintenance, incorrectly believed the year was 2006, causing a major Telstra network outage affecting voice calls, texts, emergency services, trains, and payment systems.
- Modern mobile networks, especially 5G using TDD, depend on precise time synchronization so cells switch direction in unison; inaccurate time causes self-jamming.
- Telstra's time distribution architecture had vulnerabilities: a 2020 upgrade introduced peering mode and reduced redundancy, enabling timing loops where servers took time from downstream nodes.
- Two NTP defenses (stratum weighting and outlier voting) were progressively disabled: first by peering, then by activating a GPS card that elevated a server to stratum 1 unopposed.
- The GPS receiver's firmware was outdated, causing it to assume the year 2006 after a restart due to the week number rollover, and no other source could contradict it.
- Key lessons include classifying time as critical infrastructure, documenting all changes, ensuring redundancy in both equipment and expertise, and using point-to-point connections instead of peering.
- The outage happened despite NTP working correctly, highlighting that organizational and architectural decisions were the root cause.