Bilingual

Notes on incidents
a day ago
- Most incidents involve waiting and often resolve on their own without intervention.
- Engineers often make incidents worse by jumping to quick fixes, like clearing queues or forcing redeploys.
- The first recommended action during an incident is to pause and relax, using a ritual to avoid rushing.
- Effective incident resolution actions are usually simple, such as temporarily disabling a problematic feature.
- Knowledge of the system is more valuable than having many engineers on the call.
- Resolving an incident requires courage and decisiveness, especially from those with system knowledge.
- Fixing incidents earns political credit with managers and executives, but this is not a durable position of power.
- The tendency for incidents to self-resolve may depend on system size, with larger systems having more self-recovery mechanisms.
Stamp It! All Programs Must Report Their Version
a day ago
- The author correctly guessed an outage root cause quickly but wasted hours due to lack of visibility into version numbers and rollouts.
- Software versioning standards are often lower than those of household appliances, despite software being constantly updated.
- The i3 window manager's `--version` and `--moreversion` commands provide detailed version info, including VCS revision, config file path, and build date, which aid debugging.
- Go stamps VCS buildinfo by default since Go 1.18, making it easy to identify exact build versions.
- NixOS can lose VCS revision stamping due to reproducibility concerns, but the author's `go-vcs-stamping` overlay fixes this.
- The three-step solution to improve version observability is: Stamp the VCS revision, Plumb it through build/packaging, and Report it on all relevant surfaces.
- Implementing version observability is a high-ROI one-day project that can save hours during incident response.
Hugging Face hacked: Blue Team turned to Chinese LLM after US models blocked
9 days ago
- Hugging Face's production infrastructure was breached by an autonomous AI agent system, exploiting code-execution paths to gain access and harvest credentials.
- The security team used China's open-source GLM 5.2 model for forensic analysis after commercial frontier models' safety guardrails blocked incident response efforts.
- Hugging Face recommends defenders have a vetted model ready on their own infrastructure to avoid guardrail lockout and protect data from leaving the environment.
- The breach involved lateral movement into internal clusters using short-lived sandboxes, but no tampering with models, datasets, or supply chain was found.
- The incident report coincided with the release of Chinese AI models like Kimi K3, which are cheaper and perform competitively against U.S. models.
Lessons Learned from CISA's Recent GitHub Leak
16 days ago
- CISA data leak involved contractor publishing 844 MB of sensitive data, including AWS GovCloud credentials, on a public GitHub repository for six months.
- CISA's response was delayed; it took over 48 hours to invalidate leaked keys, partly due to system complexities and unclear incident reporting channels.
- The agency's postmortem emphasizes improving key management, establishing distinct reporting channels, and continuously scanning for exposed secrets on platforms like GitHub.
- GitGuardian researcher highlighted that CISA ignored nine automated alerts, prolonging exposure, and urged organizations to make leak reporting trivial and visible.
- CISA noted enhanced logging and zero-trust principles helped limit damage, but its incident playbook lacked procedures for GitHub or cloud service incidents.
- The report is praised for transparency and advocating for secrets scanning and better relations with security researchers, setting a precedent for organizations.

incident response

Notes on incidents

Stamp It! All Programs Must Report Their Version

Hugging Face hacked: Blue Team turned to Chinese LLM after US models blocked

Lessons Learned from CISA's Recent GitHub Leak