AWS from Zero to Production — Phase 9, Part 8 of 11 · See the full series roadmap
A data platform isn’t one job. It’s dozens — sometimes hundreds — that depend on each other.
The ingestion must finish before the transform. The transform must finish before the warehouse load. The dashboard refresh waits for the load. The finance export waits for the dashboard. And when step three fails at 2 a.m., someone needs to know — and the rerun must not double-count yesterday’s revenue.
That’s orchestration: deciding what runs, when, in what order, and what happens when something fails. Cron and hope is not an orchestration strategy.
What you’ll walk away with
- 🕸️ DAGs and dependencies
- 🧭 AWS Step Functions for data pipelines
- 🌬️ Amazon Managed Workflows for Apache Airflow (MWAA)
- ⏰ EventBridge Scheduler, Glue workflows, and Lambda orchestration
- 🔁 Retries, idempotency, and backfills
- 🚨 Failure handling and notifications
- 👀 Data pipeline observability
DAGs: the shape of a pipeline
Pipelines are modeled as a directed acyclic graph (DAG): tasks connected by dependencies, flowing in one direction, with no loops.
+--> transform_orders ----+
ingest_db --+ +--> load_warehouse --> refresh_dashboards
+--> transform_customers -+
|
ingest_events --> transform_events ---------------+
- Dependencies decide order:
load_warehousewaits for all its upstream tasks. - Independent branches run in parallel.
- The orchestrator tracks the state of every task in every run.
AWS Step Functions
Step Functions (Phase 4) is a serverless workflow service — and a strong fit for AWS-centric data pipelines.
Why it works for data
- Service integrations: start Glue jobs, EMR steps, EMR Serverless jobs, Athena queries, Redshift Data API statements, Lambda functions, ECS tasks, and many more — and wait for completion with the
.syncpattern. - Built-in retries with exponential backoff, and catch blocks for failures.
- Parallel and Map states for fan-out — like processing every partition or table.
- Visual execution history — you can see exactly which step failed, and why.
- Pay per state transition (Standard workflows) — no idle servers.
{
"StartAt": "TransformOrders",
"States": {
"TransformOrders": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "raw-to-curated-orders",
"Arguments": { "--RUN_DATE.$": "$.run_date" }
},
"Retry": [{
"ErrorEquals": ["States.ALL"],
"IntervalSeconds": 60,
"MaxAttempts": 2,
"BackoffRate": 2
}],
"Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "NotifyFailure" }],
"Next": "RefreshPartitions"
},
"RefreshPartitions": {
"Type": "Task",
"Resource": "arn:aws:states:::athena:startQueryExecution.sync",
"Parameters": {
"QueryString": "MSCK REPAIR TABLE sales_curated.orders",
"WorkGroup": "pipelines"
},
"Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "NotifyFailure" }],
"End": true
},
"NotifyFailure": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"TopicArn": "arn:aws:sns:us-east-1:111122223333:data-alerts",
"Message.$": "States.Format('Pipeline failed: {}', $.Error)"
},
"Next": "Fail"
},
"Fail": { "Type": "Fail" }
}
}
Amazon MWAA (Apache Airflow)
Apache Airflow is the most widely used open-source orchestrator for data pipelines. Amazon Managed Workflows for Apache Airflow (MWAA) runs it for you.
- Pipelines are Python DAGs, stored in S3.
- Huge provider ecosystem: operators for AWS services, databases, dbt, Snowflake, Databricks, and SaaS tools.
- Strong scheduling features: catchup, backfills, data-aware scheduling with datasets or assets, and SLAs.
- A web UI for runs, logs, and retries.
from datetime import datetime
from airflow import DAG
from airflow.providers.amazon.aws.operators.athena import AthenaOperator
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
with DAG(
dag_id="orders_daily",
start_date=datetime(2026, 10, 1),
schedule="0 3 * * *",
catchup=False,
default_args={"retries": 2},
) as dag:
transform = GlueJobOperator(
task_id="transform_orders",
job_name="raw-to-curated-orders",
script_args={"--RUN_DATE": "{{ ds }}"},
)
repair = AthenaOperator(
task_id="refresh_partitions",
query="MSCK REPAIR TABLE sales_curated.orders",
database="sales_curated",
workgroup="pipelines",
)
transform >> repair
Cost note: An MWAA environment has an hourly charge while it exists, whether or not DAGs run. It’s worth it for many pipelines; it’s expensive for one.
Step Functions or MWAA?
- Step Functions: AWS-native, serverless, pay-per-use, and excellent for event-driven or moderate numbers of pipelines.
- MWAA: many pipelines, a Python-first data team, rich scheduling and backfill features, and non-AWS integrations.
Other scheduling and orchestration options
EventBridge Scheduler
A serverless scheduler for one-time or recurring invocations — cron or rate expressions, with time zones and flexible windows. Use it to start Step Functions state machines, Glue jobs, Lambda functions, and many other targets.
aws scheduler create-schedule \
--name orders-daily \
--schedule-expression "cron(0 3 * * ? *)" \
--schedule-expression-timezone "UTC" \
--flexible-time-window Mode=OFF \
--target '{"Arn":"arn:aws:states:us-east-1:111122223333:stateMachine:orders-daily","RoleArn":"arn:aws:iam::111122223333:role/scheduler-start-pipeline"}'
Glue workflows
Chain Glue crawlers and jobs with triggers (Part 3). Good when the pipeline lives entirely inside Glue.
Lambda orchestration
Calling one Lambda from another to sequence steps is tempting — and fragile. Lambda has a 15-minute limit, retries become hard to reason about, and there’s no visual state. Use Lambda for steps, and an orchestrator for the flow.
Dependencies
Two kinds:
- Task dependencies: B runs after A in the same run.
- Data dependencies: B runs when the data it needs is ready — even if it was produced by another pipeline.
Handle data dependencies with:
- Events: the upstream pipeline emits an event to EventBridge when it finishes; the downstream pipeline starts on that event.
- Sensors: Airflow tasks that wait for a file, partition, or condition.
- Dataset/asset scheduling in Airflow, where DAGs trigger when the datasets they consume are updated.
Retries
Transient failures — throttling, network blips, a briefly unavailable service — are normal. Retry them automatically:
- Exponential backoff — 1 minute, 2 minutes, 4 minutes — with jitter.
- Limited attempts — two or three, then fail loudly.
- Retry only what’s retryable — don’t retry a data validation failure five times.
Idempotency: safe to run again
An idempotent task produces the same result whether it runs once or five times. It’s the most important property of a data pipeline, because retries and reruns will happen.
Not idempotent: INSERT INTO daily_revenue SELECT ... WHERE day = '2026-10-31'
-> run twice, revenue doubles
Idempotent: DELETE FROM daily_revenue WHERE day = '2026-10-31';
INSERT INTO daily_revenue SELECT ... WHERE day = '2026-10-31'
-> or overwrite the partition, or MERGE by key
Techniques:
- Overwrite partitions instead of appending: Spark dynamic partition overwrite, Athena or Redshift delete-then-insert in a transaction.
- MERGE by a business key — with Iceberg or Redshift.
- Parameterize by run date — the task processes
run_date=2026-10-31, not “now.” - Deterministic outputs — same input, same output files and locations.
Backfill
A backfill reruns a pipeline for past dates — after fixing a bug, adding a column, or onboarding a new table.
Backfills are easy only if tasks are:
- Parameterized by date — never relying on
now(). - Idempotent — safe to overwrite.
- Independent per date — so dates can run in parallel, within limits.
Airflow has backfill built in. With Step Functions, a Map state over a list of dates — with a concurrency limit — does the same job.
Watch out: A backfill of 365 days at full parallelism can overwhelm sources, hit service quotas, and run up a big bill. Throttle it.
Failure handling
Decide in advance what happens when each step fails:
- Retry, for transient errors.
- Fail the run and alert, for real problems.
- Quarantine bad records and continue, when partial results are acceptable.
- Skip downstream tasks that depend on failed ones — never publish half-finished data.
- Clean up partial outputs, or write to a temporary location and swap atomically on success.
Notifications
People need to know — but not about everything:
- Failures → SNS → email, chat, or paging, with a link to the failed run.
- SLA misses → “the daily revenue table isn’t ready by 7 a.m.”
- Success → usually a metric or a dashboard, not an email. Success emails get ignored, and then so do failure emails.
Route Step Functions, Glue, and EMR state-change events through EventBridge to a single alerting path.
Data pipeline observability
Infrastructure metrics aren’t enough. A pipeline can “succeed” and still deliver wrong or empty data. Track:
- Run status and duration — per pipeline and per task, over time.
- Data freshness — when was each table last updated?
- Volume — row counts per run. A sudden drop to zero is a red flag.
- Quality — Glue Data Quality scores and rule failures.
- Cost — per pipeline, using tags.
Alarm ideas:
Step Functions ExecutionsFailed > 0
Glue job duration > 2x the 7-day average
orders_curated row count today < 50% of yesterday
daily_revenue not updated by 07:00 UTC
Common mistakes
- Cron jobs with no dependency tracking.
- Non-idempotent tasks that double-count on retry.
- Using
now()inside tasks, making backfills impossible. - Lambda chains as an orchestrator.
- Success emails that drown out failures.
- No freshness or volume checks — pipelines “succeed” with no data.
- Unthrottled backfills.
Key takeaways
- Pipelines are DAGs of tasks with task and data dependencies.
- Step Functions is serverless and AWS-native; MWAA brings Airflow’s ecosystem and scheduling.
- EventBridge Scheduler starts things on time; orchestrators manage the flow.
- Make every task idempotent and parameterized by date — retries and backfills depend on it.
- Plan failure handling, alert on failures and SLAs, and monitor freshness, volume, and quality.
Quick self-check
Test yourself before moving on:
- Why is
INSERT INTO ... SELECTfor “today” not idempotent? - What makes a pipeline easy to backfill?
- When would you choose MWAA over Step Functions?
- A pipeline succeeded but the table is empty. What kind of monitoring would catch it?
Answers
- Running it twice inserts the data twice. Overwrite the partition or MERGE instead.
- Tasks that are parameterized by date, idempotent, and independent per date.
- For many pipelines, a Python-first team, rich scheduling and backfills, or non-AWS integrations.
- Data observability — volume (row count) and freshness checks.
What’s next
In Part 9, we get data into the platform: AWS DMS full loads and change data capture, schema conversion, DataSync, Transfer Family, AppFlow, RDS exports, SaaS ingestion, and migration validation.
Step Functions or Airflow — which runs your pipelines? Tell me why in the comments.