Reliability and Asynchronous Work

A failure model is a list of ways a component can stop meeting its promise. A process can crash, a machine can disappear, a network can delay or drop messages, a dependency can return bad data, and a person can deploy the wrong configuration. These failures have different causes and need different defenses.

Start with one user journey and inspect every step. For each step, record the expected result, how long it may take, whether it changes data, and what the caller sees when it fails. Also identify the failure domain: one process, one machine, one availability zone, one region, or the whole organization.

For a checkout flow, the catalog may be optional while payment and order storage are critical. If recommendations fail, the page can continue. If payment is uncertain, the system must stop and reconcile rather than guess. This simple classification guides later choices about timeouts, retries, and degraded modes.

A timeout is a deadline after which the caller stops waiting. Without one, a slow dependency can hold threads, connections, memory, and user attention until the whole service becomes saturated. A timeout does not prove that remote work stopped; it only means the caller no longer waits for the result.

Choose deadlines from the end-to-end latency budget. If a page must finish in two seconds, a secondary API cannot use a five-second timeout. Connect, read, and total request timeouts may need different values. The caller must also decide what to show next: an error, cached data, a partial response, or a pending state.

Suppose an address lookup normally takes 100 ms and its p99 is 400 ms. A 700 ms timeout leaves room for normal variation without blocking checkout for several seconds. Metrics should count deadline failures so the team can tell whether the value is too strict or the dependency is unhealthy.

Retries help with transient failures such as a brief network interruption or a busy service. They do not fix permanent failures such as invalid input or missing permissions. Retrying every error wastes capacity and can turn one unhealthy service into a system-wide incident.

Exponential backoff increases the wait between attempts, for example 100 ms, 200 ms, then 400 ms. Jitter adds randomness so thousands of clients do not retry at the same instant. Always set a maximum attempt count or total retry budget, and retry at one deliberate layer rather than at every layer.

A read-only product lookup may safely retry a timeout once. A charge request is different because the first attempt may have succeeded even when its response was lost. That write needs an idempotency design before automatic retries are safe.

Idempotency matters because networks cannot always tell a caller whether a request completed. A timeout may happen before processing, during processing, or after the server committed the change but before the response arrived. The caller needs a safe way to try again without creating a second payment or order.

A common design uses an idempotency key generated for one user intent. The server stores the key with the result in the same reliable boundary as the side effect. A repeated request with the same key returns the stored result. A unique database constraint can provide the final protection against races.

When a user presses Pay, the client creates one key for that payment attempt. Network retries reuse it. A new payment intent gets a new key. The server validates that a reused key has the same request details so the key cannot hide conflicting operations.

A circuit breaker protects capacity. In the closed state, calls pass normally. After failures cross a threshold, it opens and fails fast without contacting the dependency. After a cool-down, a small number of trial calls enter a half-open state to test recovery.

The breaker must be scoped and measured carefully. One global breaker can hide a healthy region because another region is failing. A threshold that is too sensitive causes needless outages, while one that is too slow allows saturation. Breaker events need dashboards and alerts because a fast fallback can otherwise hide a long incident.

If a shipping quote provider is unavailable, an open breaker can stop hundreds of slow calls and show that exact delivery prices are temporarily unavailable. The team can still preserve browsing and carts while the provider recovers.

A degraded mode is an intentional lower level of service. Examples include hiding recommendations, serving slightly stale public data, switching to read-only mode, or accepting a request for later processing. It is different from silently returning wrong data; the remaining behavior must still be safe and honest.

Choose degradation by ranking capabilities as critical, important, or optional. Define the trigger, the user message, the data freshness allowed, and the recovery condition. Test the mode before an incident because code paths that are never exercised often fail when needed.

An event site can keep schedules readable from a recent cache while registration storage is unavailable. It must disable the registration button and explain the temporary limitation rather than pretending a booking succeeded.

The producer writes a message describing a job. A queue holds it until a consumer or worker claims it. This design shortens user requests and absorbs traffic spikes, but it introduces states such as queued, processing, succeeded, and failed. A response of 202 Accepted means work was accepted, not completed.

Most queues provide at-least-once delivery, so a worker may receive the same message more than once. Workers therefore need idempotency, bounded retries, a dead-letter path, and visibility into queue depth and oldest-message age. Message payloads should carry stable identifiers, not large mutable objects.

For a ninety-second report, the API creates one job and returns its ID. A worker builds the file, stores the result, and updates job status. The UI polls or receives a notification and shows failure with a retry option instead of keeping one HTTP request open.

Begin with the failure model and a latency budget. Use a timeout where waiting must end. Add a retry only for transient errors, and require idempotency before retrying uncertain writes. Use a circuit breaker when repeated calls can exhaust capacity, and define graceful degradation for optional work.

If work is slow or bursty, place it behind a queue and expose honest job states. The worker uses the same idempotency rule, bounded retries, and dead-letter handling. Metrics connect the design: request latency, breaker state, queue age, job failure rate, and the user outcome all need visible thresholds.

Consider image processing after an upload. The API validates and stores the original once, creates one idempotent job, and returns a job ID. Workers generate variants, retry temporary storage errors with backoff, and mark permanent failures for support. The UI can still show the original while processing continues.