Microsoft Azure Learning Journey

Design Azure Systems That Handle Partial Failure

Keep distributed APIs correct when requests time out, messages repeat, dependencies slow down, and multi-step work cannot be rolled back atomically.

5 min read · 824 words
Microsoft AzureCloud ArchitectureDistributed SystemsReliability EngineeringDevOps
Design Azure Systems That Handle Partial Failure cover image

A request can time out after the server commits the change. A message can be delivered again after a worker restarts. One service can succeed while the next is unavailable. These are normal distributed-system states, not exceptional edge cases. A robust Azure architecture defines what the caller can retry, which action is idempotent, how long work may wait, and how a partially completed business process recovers.

What you’ll learn

Prerequisites

Treat a timeout as an unknown outcome

When a client times out, it usually cannot know whether the server committed the operation. Retrying a non-idempotent “charge card” or “create order” request can create a second effect. Give operations a stable idempotency key and persist the key/result with the business change in the same consistency boundary. Reject reuse of a key with a different payload.

For each dependency call, set a deadline that fits the user-facing request budget. Retry only transient failures, cap attempts, add exponential backoff and jitter, and avoid retrying at every layer. A client, API gateway, service, and SDK that each retry three times can multiply a single request into many backend calls. Use a circuit breaker or bulkhead to contain repeated failures and define a safe fallback.

Handle multi-service work as a workflow

A saga coordinates a sequence of local transactions across services. Each step has a durable state, retry policy, and, where possible, a compensating action. Compensation is a new business action—not a database rollback. A refund may reverse a payment effect, but it does not erase the fact that the original payment happened.

The transactional outbox addresses the gap between committing a local database change and publishing its event. Write the business change and an outbox record in the same local transaction; a dispatcher publishes pending records and marks progress. The dispatcher can publish duplicates, so consumers still need idempotency. The outbox is not a distributed transaction across arbitrary databases and brokers.

Persist workflow state and make timeout, cancellation, compensation failure, and manual intervention visible. Avoid a chain of synchronous calls that holds a user request open while independent services wait on each other.

Put API Management in its proper role

Azure API Management can provide a managed API gateway and API lifecycle capabilities: routing, authentication integration, quotas/rate limits, transformations, and developer-facing products, depending on the tier and configuration. It does not make a backend operation idempotent, guarantee its database consistency, or replace application authorization.

Use stable API versions and document compatibility. Validate caller identity and authorization at the appropriate boundary; do not treat an API key as proof of user identity. Apply throttling before expensive downstream work and return a clear retry signal when the caller can safely retry. Protect backend endpoints so they cannot bypass required gateway controls unless that direct route is intentionally supported.

Apply backpressure and isolate failures

Use bounded queues and concurrency. When a downstream dependency slows, reduce intake, shed optional work, or return an explicit overload response instead of building an unbounded backlog. Separate workloads with different latency or failure characteristics into their own pools or queues. Track queue age and oldest-message time, not just queue length.

Use asynchronous workflows when the operation does not need to finish inside the request. Return an operation ID and a status endpoint or notification channel. Define expiry, cancellation, and recovery semantics. If eventual consistency is part of the design, state the expected delay and how consumers learn that data is ready.

Give each service a data boundary

Services that share a database also share schema, availability, and deployment coupling. Prefer ownership boundaries where one service controls writes and exposes an API, event, or read model to other services. A deliberate read-only replica or analytical copy can be useful; hidden cross-service writes undermine the boundary.

✅ Checkpoint: For a request that creates an order and charges a payment service, specify the idempotency key, timeout budget, retry owner, outbox state, saga compensation, user-visible pending state, and operator path for an unrecoverable step.

Next: Plan and test Azure recovery.

Series navigation: Previous: Event-driven architecture · Current: Phase 10, Part 2 · Next: High availability and disaster recovery

References

Share this article

LinkedIn post draft