Retries don't fix eventual consistency

6 hours ago

Retries are not always the correct solution; they can be applied to problems that are not actual failures.
In distributed systems, events arriving out of order is a normal property of eventual consistency, not a bug.
A proposed system treated missing 'user' data when a 'subscription' event arrived as an error requiring retries, but this was a misidentification of the issue.
Eventual consistency means information will arrive eventually, so missing data is just another valid system state, not an exceptional one.
The recommended approach is to store all incoming data and only process when all prerequisites are met, avoiding retries and dead-letter queues.
Retrying or replaying messages can create additional problems like ordering and infrastructure complexity that wouldn't exist otherwise.
Retries are still useful for transient failures like network timeouts, but repeated retries should be questioned for their efficacy.
The core lesson is to correctly identify the problem type—availability vs. eventual consistency—before choosing a solution tool.
Solving the actual problem reduces operational overhead and makes systems easier to reason about, even if it means not retrying at all.

The incident involved intermittent failures in a search backend triggered by a slow dependency, not CPU throttling.
The false root cause (CPU throttling) was initially appealing but contradicted by topology, timing, and failure distribution evidence.
The actual cause was a combination of no timeouts on downstream calls and eager retries, creating a feedback loop that saturated the worker pool.

Junior researchers struggle with long-term problems, needing to break 400-hour problems into smaller 1-hour ones.
Subproblems can be created by asking simpler questions, studying simpler systems, or fact-finding.
Toy problems and limits help simplify complex problems for easier analysis.

A check that has never failed is unproven—break the thing deliberately to confirm it can fail.
Check at the layer where the consequence happens (e.g., rendered page, not just source) because easy checks miss failures.
Reliability is not validity: a measure can be consistent but measure the wrong thing; test invariance under irrelevant transformations.

The report 'Patiëntveiligheid bij ICT-uitval in ziekenhuizen' emphasizes that ICT failures in hospitals can be life-threatening.
High reliability in ICT requires simple systems with independent emergency solutions, not complex redundancy.
Complex redundancy often leads to larger and longer outages due to increased complexity (e.g., network loops, multi-master database conflicts).

Business logic often follows 'when something happens, something else should happen' pattern.
Coupling facts and consequences in a single script obscures the domain fact and creates maintenance issues.
Domain events record facts in past tense, not commands; they are immutable and contain enough information to identify the fact.

The incident involved intermittent failures in a search backend triggered by a slow dependency, not CPU throttling.
The false root cause (CPU throttling) was initially appealing but contradicted by topology, timing, and failure distribution evidence.
The actual cause was a combination of no timeouts on downstream calls and eager retries, creating a feedback loop that saturated the worker pool.

Junior researchers struggle with long-term problems, needing to break 400-hour problems into smaller 1-hour ones.
Subproblems can be created by asking simpler questions, studying simpler systems, or fact-finding.
Toy problems and limits help simplify complex problems for easier analysis.

A check that has never failed is unproven—break the thing deliberately to confirm it can fail.
Check at the layer where the consequence happens (e.g., rendered page, not just source) because easy checks miss failures.
Reliability is not validity: a measure can be consistent but measure the wrong thing; test invariance under irrelevant transformations.

The report 'Patiëntveiligheid bij ICT-uitval in ziekenhuizen' emphasizes that ICT failures in hospitals can be life-threatening.
High reliability in ICT requires simple systems with independent emergency solutions, not complex redundancy.
Complex redundancy often leads to larger and longer outages due to increased complexity (e.g., network loops, multi-master database conflicts).

Business logic often follows 'when something happens, something else should happen' pattern.
Coupling facts and consequences in a single script obscures the domain fact and creates maintenance issues.
Domain events record facts in past tense, not commands; they are immutable and contain enough information to identify the fact.

Hasty Briefsbeta