A scheduling tool times out after receiving a request. The agent has enough information to say that the call failed. It does not have enough information to say whether the appointment exists.
That difference determines the reliability work. Faster retries could create duplicate appointments. A durable action record and a read-back path could turn the ambiguous result into a confirmed outcome. Improving availability and improving reliability point to different investments.
Price the consequence
I start with the user’s work. A brief failure before an irreversible action may be harmless if the system retries safely. A successful response can be severe if it claims an action completed when no external effect occurred. The same duration and error rate can produce very different consequences.
I describe each important failure using four fields:
| Field | Question |
|---|---|
| Outcome | Which user result failed? |
| Exposure | How often can the condition occur, and for whom? |
| Consequence | What must the user or operator do next? |
| Recovery | Can the system detect, contain, and repair the failure? |
This keeps reliability tied to a decision. A high-frequency failure with cheap automatic recovery may deserve less work than a rare silent error that changes external state.
Choose where to spend
Reliability effort can prevent a failure, detect it, limit its effect, or recover afterward. Prevention is one option among several.
For a tool call with uncertain completion, preventing every timeout may be unrealistic. An idempotency key can limit duplicate effects. An external read can confirm the result. A durable action record can support repair. The combination may produce a better user outcome than pursuing a network path that never fails.
I compare the options against the same consequence:
- How much does this control reduce exposure?
- Which new failure does it introduce?
- How quickly will we know whether it worked?
- Who operates it when the original builder is absent?
- What product speed or system cost does it consume?
The best control often combines a modest reduction in failure probability with a large reduction in consequence.
Set an operating bar
A reliability target should force a release decision. “Improve recovery” does not. “For supported tasks, every accepted action ends in a confirmed effect, a confirmed absence of effect, or a transfer with the action state attached” does.
The bar can include several measurements, but each one needs a role. One metric may guard user harm. Another may cap operating cost. A third may indicate that the workload has moved outside the tested regime.
I record the accepted residual risk. If the team ships with a known failure class, the decision should name its expected frequency, consequence, detection path, and owner. Hidden acceptance is still acceptance; it simply cannot be managed.
Spend from evidence
Reliability work attracts hypothetical edge cases. Some reveal real structural weaknesses. Others consume the budget without changing observed risk.
I rank the next work using production evidence, adversarial tests, and consequence. An unobserved failure with severe impact may still deserve action when the mechanism is credible. The decision record should explain why its modeled risk outranks measured failures.
After a control ships, I check whether it changed the intended measure. If retries rise while completed tasks do not, the work moved activity instead of outcome. If a fallback reduces user harm but doubles manual repair, the cost moved to operations and belongs in the evaluation.
Reallocate the budget
The allocation should change as the system gains users, authority, and dependencies. A low-risk assistant can tolerate behavior that becomes unacceptable once it may change an account or place an order.
I revisit the budget when:
- the system gains a new consequential action;
- traffic or user mix moves beyond the tested cohort;
- an operator becomes a hidden recovery dependency;
- a control costs more than the failure it addresses;
- new evidence changes the estimated consequence.
I expect the final allocation to name each control, the failure it addresses, its expected effect, and its operating owner. It should also show the first unfunded risk. That is where the next reliability discussion begins.