Horror Stories from Former Azure Engineer
4 hours ago
- #Azure Security
- #Technical Debt
- #VM Density
- Microsoft's Azure team attempted to increase VM density from 32 to 48 per node, leading to a 50% rise in crashes and incidents, revealing scalability issues.
- The Hyper-V team observed high call volumes to its WMI interface (up to 10,000 calls per second) without clear visibility, making offloading efforts like Overlake impractical.
- The WireServer (instance metadata service) on the host OS posed significant security risks by exposing unsecured endpoints to guest VMs, potentially allowing VM escapes or data breaches.
- WireServer caches contained unencrypted tenant data in violation of security guidelines and suffered from memory leaks and crashes (300,000-500,000 per month).
- Technical leadership gaps hindered refactoring and fixes, with defensive responses from leadership when security concerns were raised.