When Someone Else’s Outage Stops Your Business: Designing Systems That Survive Supplier Failure

Ask an engineer how many external services their system depends on and you will usually get a number. Ask them to list all of them, and the number grows while they talk.

A modern business system is not one piece of software. It is a payment provider, a mail delivery API, an address validation service, an ERP connector, an identity provider, a CDN, a cloud region, a DNS resolver, and half a dozen smaller APIs that someone integrated years ago for a specific need. Each one works almost all of the time. None of them belongs to you.

That is the reality worth designing for, and most processes are not designed for it at all.

The process everybody designs

Walk through how a typical order, claim or onboarding process is specified and you will find a clean sequence: the customer submits, the address is validated, the payment is authorised, the record is created in the ERP, the confirmation email goes out, the document is archived.

Every step assumes the previous one succeeded. Every external call assumes the provider answers. The design describes what happens when everything works, because that is the version everyone can agree on, and because the alternative involves asking uncomfortable questions about who is responsible when it does not.

The implicit assumption is that these providers are effectively part of your system: always available, always the same, never wrong. They are not. They are independent companies with their own deploys, their own incidents and their own bad Tuesdays.

What actually happens

The record on this is not ambiguous. Between August 2024 and August 2025, AWS, Azure and Google Cloud together logged more than a hundred service outages. Around 94% of enterprise services worldwide rely on at least one major cloud provider, and the three largest control roughly 62% of the market. Concentration is efficient right up until it is not.

The individual incidents are instructive, because none of them were exotic:

In October 2025, a DNS resolution failure in a single AWS region took services down for hours, affecting more than 3,500 companies across 60 countries. The trigger was an empty DNS record and an automation update.
In November 2025, a file larger than expected fed into Cloudflare’s Bot Management system propagated worldwide within minutes, degrading service for a large share of the 2.4 billion users whose traffic passes through that network. Estimated cost: upwards of 250 million dollars.
In July 2024, a faulty security agent update crashed Windows hosts globally. It was not a cloud failure at all, and analysts put the damage to Fortune 500 companies alone at 5.4 billion dollars.
In May 2026, invalid DNSSEC signatures published for the .de zone made domains unresolvable across the internet, including second-level domains that did not use DNSSEC themselves.

That last one deserves a second reading. Companies were taken offline by a dependency they had never configured, never chosen and probably never listed. This is the shape of the problem: the dependencies that hurt are rarely the ones you are watching.

There is one more detail worth knowing. Analysis of recent major incidents found that roughly a quarter of total customer impact time elapsed before the provider itself knew what had broken. Waiting for the status page to tell you what is happening is not a recovery strategy.

When the process stops, something else stops too

Here is where this becomes a business conversation rather than a technical one.

If a third-party call fails and the process halts, the consequence depends entirely on what that process does. A marketing page being slow is an annoyance. An order pipeline that cannot accept orders is lost revenue, and it does not come back later. A logistics system that cannot produce shipping labels stops trucks. A billing run that cannot complete on the last day of the month becomes a cash flow problem and a reconciliation problem. In regulated contexts, a process that stops can become a reporting obligation.

The important question is not “how reliable is this provider”. It is: if this provider is unavailable for four hours, what stops, and what does that cost us?

Most companies have never asked it per dependency. They have a general sense that an outage would be bad, which is not specific enough to design against.

What a resilient design actually looks like

Resilience is not about achieving perfect uptime, which you cannot buy at any price when the failure is upstream. It is about making sure that a supplier failure degrades your process instead of stopping it.

A few patterns do most of the work.

Know the full dependency map

You cannot design around what you have not listed. That means every external call, including the ones buried in a library, inherited from a vendor module, or added years ago by someone who has since left. It also means the dependencies underneath your dependencies: the region, the DNS, the CDN, the identity provider.

This mapping is the unglamorous first step, and it is where most of the surprises turn up.

Decide, per dependency, what failure should do

Not every dependency deserves the same treatment. For each one, there are only a few sensible answers:

Fail the transaction, when proceeding without it would be wrong. A payment authorisation cannot be assumed.
Degrade gracefully, when the step is enriching rather than essential. If address validation is down, accept the address and flag it for review rather than blocking the order.
Queue and retry, when the step can happen later without harm. Confirmation emails and downstream syncs rarely need to be synchronous, and treating them as such converts a provider hiccup into a customer-facing failure.
Switch to an alternative, when the function is genuinely critical and a second provider exists. This is the most expensive option, which is why it should be a deliberate choice for a small number of dependencies rather than an aspiration for all of them.

The point is that each of these is a business decision expressed in code. Someone has to decide whether an order should be accepted when a non-essential check is unavailable. Engineering can implement either answer, but it should not be the one guessing.

Contain the failure

Timeouts, circuit breakers and bulkheads exist so that a slow provider does not become a stopped system. Without a timeout, a hanging external call holds a connection, then a thread, then the pool, and a single degraded integration takes down an application that had many other things to do. A surprising share of full outages start as one slow dependency.

Make the queue the default for anything asynchronous

If a step does not need to complete before the user gets an answer, it should not be in the critical path. Putting it behind a durable queue means a provider outage delays work rather than losing it, and the process catches up by itself when the provider returns.

Rehearse it

A fallback path that has never been executed is a hypothesis. The cheapest useful test is to disable a dependency in a staging environment and watch what the process does. The answers are often uncomfortable, which is exactly why the exercise is worth doing before the provider chooses the date for you.

This is a design question, not a procurement question

It is tempting to treat supplier risk as something to solve in contracts. SLAs and credits matter, but they compensate you after the fact. They do not keep orders flowing on the morning a DNS record goes wrong somewhere you do not control.

The thing that keeps the process running is a design that anticipated the failure. And that design is only possible if someone understands the process end to end: not just the code, but what each step is for, which ones the business genuinely cannot proceed without, and what an acceptable degraded mode looks like.

That whole-process view is what we produce in a technology assessment, and what we maintain over time when we take over a system. It is also why we keep insisting that documentation is not an administrative nicety: you cannot design resilience into a process nobody can fully describe.

Perfect uptime is not available. A process that survives its suppliers is.

Frequently asked questions

What is system resilience in a business context?
Resilience is the ability of a business process to keep running when parts of it fail, particularly parts you do not control. It is not the same as uptime. A resilient design accepts that external providers will fail and ensures the process degrades, queues or reroutes rather than stopping.

Why do outages at providers like AWS or Cloudflare affect us if we are not their direct customer?
Because dependencies are layered. Your provider may run on a cloud region, behind a CDN, using a DNS resolver or identity service you never selected. In May 2026, invalid DNSSEC signatures on the .de zone made domains unresolvable even for organisations that had not enabled DNSSEC themselves. The dependency map is usually deeper than the contract list.

Do we need a second provider for everything?
No, and trying is usually a poor investment. Redundancy is expensive in build time and in ongoing maintenance. The useful approach is to decide per dependency: fail, degrade, queue and retry, or switch. Full redundancy is worth it only for the few dependencies where the process genuinely cannot stop.

How do we find out where our real risk is?
Map every external dependency, including indirect ones, then ask a single question for each: if this is unavailable for four hours, what stops and what does it cost. That turns a vague fear of outages into a short, prioritised list of things worth engineering around.

Is this something we fix once?
No. Dependencies change as integrations are added, vendors change their terms, and providers deprecate services. Resilience is a property of a system that is maintained, not a project that concludes.

Not sure what your systems actually depend on, or what would stop if a provider went down tomorrow? Start with a fixed-scope technology assessment, or get in touch.