A request can time out after the server commits the change. A message can be delivered again after a worker restarts. One service can succeed while the next is unavailable. These are normal distributed-system states, not exceptional edge cases. A robust Azure architecture defines what the caller can retry, which action is idempotent, how long work may wait, and how a partially completed business process recovers.
What you’ll learn
- Design idempotent operations, retries, and consistency boundaries.
- Use sagas and transactional outbox patterns without promising global transactions.
- Apply Azure API Management for API lifecycle and policy enforcement.
- Control backpressure, throttling, timeouts, and failure isolation.
- Reduce coupling by giving services ownership of their data.
Prerequisites
- Basic API, database transaction, messaging, and Azure identity concepts.
- No subscription is required for the design exercise.
- API gateways, messaging, duplicated state, and high availability can add cost and operating complexity.
Treat a timeout as an unknown outcome
When a client times out, it usually cannot know whether the server committed the operation. Retrying a non-idempotent “charge card” or “create order” request can create a second effect. Give operations a stable idempotency key and persist the key/result with the business change in the same consistency boundary. Reject reuse of a key with a different payload.
For each dependency call, set a deadline that fits the user-facing request budget. Retry only transient failures, cap attempts, add exponential backoff and jitter, and avoid retrying at every layer. A client, API gateway, service, and SDK that each retry three times can multiply a single request into many backend calls. Use a circuit breaker or bulkhead to contain repeated failures and define a safe fallback.
Handle multi-service work as a workflow
A saga coordinates a sequence of local transactions across services. Each step has a durable state, retry policy, and, where possible, a compensating action. Compensation is a new business action—not a database rollback. A refund may reverse a payment effect, but it does not erase the fact that the original payment happened.
The transactional outbox addresses the gap between committing a local database change and publishing its event. Write the business change and an outbox record in the same local transaction; a dispatcher publishes pending records and marks progress. The dispatcher can publish duplicates, so consumers still need idempotency. The outbox is not a distributed transaction across arbitrary databases and brokers.
Persist workflow state and make timeout, cancellation, compensation failure, and manual intervention visible. Avoid a chain of synchronous calls that holds a user request open while independent services wait on each other.
Put API Management in its proper role
Azure API Management can provide a managed API gateway and API lifecycle capabilities: routing, authentication integration, quotas/rate limits, transformations, and developer-facing products, depending on the tier and configuration. It does not make a backend operation idempotent, guarantee its database consistency, or replace application authorization.
Use stable API versions and document compatibility. Validate caller identity and authorization at the appropriate boundary; do not treat an API key as proof of user identity. Apply throttling before expensive downstream work and return a clear retry signal when the caller can safely retry. Protect backend endpoints so they cannot bypass required gateway controls unless that direct route is intentionally supported.
Apply backpressure and isolate failures
Use bounded queues and concurrency. When a downstream dependency slows, reduce intake, shed optional work, or return an explicit overload response instead of building an unbounded backlog. Separate workloads with different latency or failure characteristics into their own pools or queues. Track queue age and oldest-message time, not just queue length.
Use asynchronous workflows when the operation does not need to finish inside the request. Return an operation ID and a status endpoint or notification channel. Define expiry, cancellation, and recovery semantics. If eventual consistency is part of the design, state the expected delay and how consumers learn that data is ready.
Give each service a data boundary
Services that share a database also share schema, availability, and deployment coupling. Prefer ownership boundaries where one service controls writes and exposes an API, event, or read model to other services. A deliberate read-only replica or analytical copy can be useful; hidden cross-service writes undermine the boundary.
✅ Checkpoint: For a request that creates an order and charges a payment service, specify the idempotency key, timeout budget, retry owner, outbox state, saga compensation, user-visible pending state, and operator path for an unrecoverable step.
Next: Plan and test Azure recovery.
Series navigation: Previous: Event-driven architecture · Current: Phase 10, Part 2 · Next: High availability and disaster recovery