Stop Using Vulnerability Counts to Measure Software Security
13 days ago
- #Software Metrics
- #Vulnerability Management
- #Cybersecurity
- Vulnerability fix count is not the same as vulnerability count; more fixes don't necessarily mean less secure software.
- Using vulnerability fix counts to measure security is flawed and ignores nuances like discovery effort and technological advancements.
- The 'Two Airplane Mechanics' analogy illustrates the importance of transparency and documented fixes over superficial assurances.
- False comparisons in cybersecurity, such as comparing vulnerability counts between systems, are problematic and misleading.
- Vulnerability fix counts are sometimes used to justify new security practices, leading to a focus on already well-addressed issues.
- The 'Mousetrap Paradox' highlights that a fixed vulnerability is both a past mistake and a current improvement.
- Cybersecurity is inherently retrospective; security is only confirmed after vulnerabilities are found and addressed.
- Punishing engineers for admitting mistakes creates a harmful incentive to hide errors rather than improve processes.
- Proposing 'Number of Vulnerabilities Reported' (NumVulnsReported) as a denominator for security metrics to focus on discovery effectiveness.
- Introducing 'Vulnerability Recidivism Metrics' to measure repeated vulnerabilities, indicating potential process failures.
- The White House report emphasizes the difficulty of measuring software security and the need for safe environments to admit and learn from mistakes.