Availability and recovery budget · Tools

The chain

Every hard dependency multiplies. Nothing adds.

List what a request has to touch to succeed. A hard dependency fails the request when it fails. A soft one only degrades it, because there is a cache, a queue, or a fallback that has actually been tested. Copies are independent instances serving at the same time, in separate zones or regions. The figures loaded below are examples: replace them with the SLA for your SKU and configuration, or better, with the availability you measured.

Example Target %

Two copies only count if they fail independently. Same region, same deploy, same identity provider: that is one copy.

The restore

The RTO is what the restore takes, not what the slide says.

Backups tell you the data exists somewhere. They tell you nothing about how long it takes to be a system again. List the steps between "it's down" and "it's back", in order, with the minutes each one takes. Mark the steps that run alongside the one above. Then tick only the steps somebody has actually timed, on purpose, in a test. The steps loaded below are the timeline I see most often; the minutes are examples.

Stated RTO, hours Stated RPO, minutes Backup every, minutes

"Alongside" means the step runs at the same time as the one above it, so only the longer of the two counts.

The note

Both numbers, on one page, ready for the design review.

This updates as you type. Paste it into the design document or the decision record, next to the name of whoever owns each number. The link button saves your chain and your restore steps into a URL, so you can send the exact version to a colleague.

From the work

Why I stopped trusting the posture people describe.

Every rule built into this tool came from a room where the number on the slide and the number in the system turned out to be different things.

A database escalation

Three confident answers. None matched the system.

During a database escalation I asked what the resiliency posture was. Three people answered, all confident, none agreeing with each other, and none matching what was actually configured. Each carried the version from the last time they touched the system.

Now: I write down, in one place, which failover is configured and tested, which is configured and never tested, and which is a slide that never got built. That is why this tool separates "copies" from hopes.

An assessment with 32 gaps

The first thing it restored depended on the fourth.

One of the gaps in that assessment was a DR plan that had never been exercised against the real dependency map. On paper it was fine. In practice the restore order was wrong, and nobody had noticed because nobody had tried.

Now: restore one system a month into an isolated environment, and time it. If the restore takes longer than the RTO on the slide, the slide is wrong. That is the "timed" column.

Regions nobody chose

Failover only works to somewhere you have set up.

In another escalation I asked which regions were real options for the workload. Nobody could say. The primary had been chosen by whoever created the first subscription and the secondary was whatever the platform assigned.

Now: a second copy in this tool only counts if the region it runs in was decided on purpose: primary, secondary, why, and what data is allowed to leave.

My own platform on AWS

The restore test is the job I like least.

On the platform I run alone, the monthly restore test is the least enjoyable thing I do. It is also the only reason I am confident the RTO in my own runbook is true.

Now: a recovery number nobody has measured is a rumour with an approval signature on it. Measure it, date it, and measure it again after anything changes.

Assumed posture and actual posture drift apart quietly. The only way to know the gap is to write it down and test it.
Khalid Shams

How the maths works

Six questions, and the assumption under each answer.

The arithmetic is simple. The judgment is in deciding what counts as hard, what counts as a copy, and which number you trust.

Why multiply, rather than take the lowest number?

Because a request needs every hard dependency to be up at the same moment. Four components at 99.99% each give about 99.96% together, not 99.99%. The chain is always worse than its weakest link, and every dependency you add makes it worse again. This is the single most common mistake I see in a design review: quoting the database SLA as the system SLA.

What does a second copy buy, and when is it a lie?

Two independent copies fail together only when both fail, so the unavailability squares: 1 minus (1 minus A) to the power of the number of copies. Two copies at 99.95% give about 99.99998%. That only holds if the copies fail independently. Two instances in the same region, behind the same load balancer, deployed by the same pipeline at the same time, are closer to one copy. So is anything that depends on a single identity provider or a single DNS zone. And failover that has never been exercised is not a copy at all.

What makes a dependency soft?

A fallback that has been tested, and a degraded experience the business has agreed to accept. A cache in front of a catalogue, a queue in front of an email provider, a "try again later" on a recommendations panel. Turning a hard dependency soft is often the cheapest availability you can buy, cheaper than a second region. But a fallback that has never run in anger is a hard dependency with a comment on it.

Is an SLA the same as availability?

No. An SLA is a commitment with service credits attached: it tells you what you get back when the provider misses, not how often it will. It usually depends on configuration, so the same service can carry different SLAs by tier and by zone or region setup. Use the published SLA for your exact SKU as a floor for planning, and your own measured availability wherever you have it.

Why is the composite rounded down?

Because a number that gets rounded up in a design review gets quoted in a contract. The tool truncates the composite to three decimal places and never rounds it up. If the honest answer is 99.949%, it is not 99.95%.

How long is a month here?

A year of 365 days is 525,600 minutes. A month is a twelfth of that, 43,800 minutes, so the monthly figure lines up with the yearly one. Provider SLAs are usually measured per calendar month, so check how yours defines it.

Reference

The nines, in minutes.

What each availability level allows you to be down for. Keep this next to any conversation where someone says "five nines" without a budget attached. The line for 99.95% is highlighted because it is where most single-region designs land, and where most targets get set.

Availability	Per year	Per month	Per week	Per day
99%	3d 15h 36m	7h 18m	1h 40m	14m 24s
99.5%	1d 19h 48m	3h 39m	50m 24s	7m 12s
99.9%	8h 45m	43m 48s	10m 05s	1m 26s
99.95%	4h 22m	21m 54s	5m 02s	43s
99.99%	52m 34s	4m 23s	1m 00s	9s
99.995%	26m 17s	2m 11s	30s	4s
99.999%	5m 15s	26s	6s	1s

At 99.99% a month allows four and a half minutes. That is less time than most teams take to notice, page someone, and open the runbook. A target above what your on-call process can physically deliver is not a target.

Where this goes next

Put the two numbers in the design, with an owner and a date.

The composite belongs in the architecture decision record. The restore time belongs in the runbook, next to the date it was last measured. Both get reopened when a dependency is added, a region changes, or a restore test comes back slower. The playbooks show where this sits in a real build.