AWS Learning Journey

Who Runs the Pipeline at 2 A.M.? Data Orchestration on AWS

Step Functions, Amazon MWAA (Apache Airflow), EventBridge Scheduler, Glue workflows, and Lambda — plus DAGs, dependencies, retries, idempotency, backfills, failure handling, notifications, and pipeline observability.

8 min read · 1512 words
Data EngineeringApache AirflowAWSData PipelinesWorkflow Automation
Who Runs the Pipeline at 2 A.M.? Data Orchestration on AWS cover image

AWS from Zero to Production — Phase 9, Part 8 of 11 · See the full series roadmap


A data platform isn’t one job. It’s dozens — sometimes hundreds — that depend on each other.

The ingestion must finish before the transform. The transform must finish before the warehouse load. The dashboard refresh waits for the load. The finance export waits for the dashboard. And when step three fails at 2 a.m., someone needs to know — and the rerun must not double-count yesterday’s revenue.

That’s orchestration: deciding what runs, when, in what order, and what happens when something fails. Cron and hope is not an orchestration strategy.

What you’ll walk away with


DAGs: the shape of a pipeline

Pipelines are modeled as a directed acyclic graph (DAG): tasks connected by dependencies, flowing in one direction, with no loops.

            +--> transform_orders ----+
ingest_db --+                         +--> load_warehouse --> refresh_dashboards
            +--> transform_customers -+
                                                  |
ingest_events --> transform_events ---------------+

AWS Step Functions

Step Functions (Phase 4) is a serverless workflow service — and a strong fit for AWS-centric data pipelines.

Why it works for data

{
  "StartAt": "TransformOrders",
  "States": {
    "TransformOrders": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "raw-to-curated-orders",
        "Arguments": { "--RUN_DATE.$": "$.run_date" }
      },
      "Retry": [{
        "ErrorEquals": ["States.ALL"],
        "IntervalSeconds": 60,
        "MaxAttempts": 2,
        "BackoffRate": 2
      }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "NotifyFailure" }],
      "Next": "RefreshPartitions"
    },
    "RefreshPartitions": {
      "Type": "Task",
      "Resource": "arn:aws:states:::athena:startQueryExecution.sync",
      "Parameters": {
        "QueryString": "MSCK REPAIR TABLE sales_curated.orders",
        "WorkGroup": "pipelines"
      },
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "NotifyFailure" }],
      "End": true
    },
    "NotifyFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "arn:aws:sns:us-east-1:111122223333:data-alerts",
        "Message.$": "States.Format('Pipeline failed: {}', $.Error)"
      },
      "Next": "Fail"
    },
    "Fail": { "Type": "Fail" }
  }
}

Amazon MWAA (Apache Airflow)

Apache Airflow is the most widely used open-source orchestrator for data pipelines. Amazon Managed Workflows for Apache Airflow (MWAA) runs it for you.

from datetime import datetime

from airflow import DAG
from airflow.providers.amazon.aws.operators.athena import AthenaOperator
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator

with DAG(
    dag_id="orders_daily",
    start_date=datetime(2026, 10, 1),
    schedule="0 3 * * *",
    catchup=False,
    default_args={"retries": 2},
) as dag:
    transform = GlueJobOperator(
        task_id="transform_orders",
        job_name="raw-to-curated-orders",
        script_args={"--RUN_DATE": "{{ ds }}"},
    )
    repair = AthenaOperator(
        task_id="refresh_partitions",
        query="MSCK REPAIR TABLE sales_curated.orders",
        database="sales_curated",
        workgroup="pipelines",
    )
    transform >> repair

Cost note: An MWAA environment has an hourly charge while it exists, whether or not DAGs run. It’s worth it for many pipelines; it’s expensive for one.

Step Functions or MWAA?


Other scheduling and orchestration options

EventBridge Scheduler

A serverless scheduler for one-time or recurring invocations — cron or rate expressions, with time zones and flexible windows. Use it to start Step Functions state machines, Glue jobs, Lambda functions, and many other targets.

aws scheduler create-schedule \
  --name orders-daily \
  --schedule-expression "cron(0 3 * * ? *)" \
  --schedule-expression-timezone "UTC" \
  --flexible-time-window Mode=OFF \
  --target '{"Arn":"arn:aws:states:us-east-1:111122223333:stateMachine:orders-daily","RoleArn":"arn:aws:iam::111122223333:role/scheduler-start-pipeline"}'

Glue workflows

Chain Glue crawlers and jobs with triggers (Part 3). Good when the pipeline lives entirely inside Glue.

Lambda orchestration

Calling one Lambda from another to sequence steps is tempting — and fragile. Lambda has a 15-minute limit, retries become hard to reason about, and there’s no visual state. Use Lambda for steps, and an orchestrator for the flow.


Dependencies

Two kinds:

Handle data dependencies with:


Retries

Transient failures — throttling, network blips, a briefly unavailable service — are normal. Retry them automatically:


Idempotency: safe to run again

An idempotent task produces the same result whether it runs once or five times. It’s the most important property of a data pipeline, because retries and reruns will happen.

Not idempotent:  INSERT INTO daily_revenue SELECT ... WHERE day = '2026-10-31'
                 -> run twice, revenue doubles

Idempotent:      DELETE FROM daily_revenue WHERE day = '2026-10-31';
                 INSERT INTO daily_revenue SELECT ... WHERE day = '2026-10-31'
                 -> or overwrite the partition, or MERGE by key

Techniques:


Backfill

A backfill reruns a pipeline for past dates — after fixing a bug, adding a column, or onboarding a new table.

Backfills are easy only if tasks are:

Airflow has backfill built in. With Step Functions, a Map state over a list of dates — with a concurrency limit — does the same job.

Watch out: A backfill of 365 days at full parallelism can overwhelm sources, hit service quotas, and run up a big bill. Throttle it.


Failure handling

Decide in advance what happens when each step fails:


Notifications

People need to know — but not about everything:

Route Step Functions, Glue, and EMR state-change events through EventBridge to a single alerting path.


Data pipeline observability

Infrastructure metrics aren’t enough. A pipeline can “succeed” and still deliver wrong or empty data. Track:

Alarm ideas:
  Step Functions ExecutionsFailed > 0
  Glue job duration > 2x the 7-day average
  orders_curated row count today < 50% of yesterday
  daily_revenue not updated by 07:00 UTC

Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. Why is INSERT INTO ... SELECT for “today” not idempotent?
  2. What makes a pipeline easy to backfill?
  3. When would you choose MWAA over Step Functions?
  4. A pipeline succeeded but the table is empty. What kind of monitoring would catch it?

Answers

  1. Running it twice inserts the data twice. Overwrite the partition or MERGE instead.
  2. Tasks that are parameterized by date, idempotent, and independent per date.
  3. For many pipelines, a Python-first team, rich scheduling and backfills, or non-AWS integrations.
  4. Data observability — volume (row count) and freshness checks.

What’s next

In Part 9, we get data into the platform: AWS DMS full loads and change data capture, schema conversion, DataSync, Transfer Family, AppFlow, RDS exports, SaaS ingestion, and migration validation.

Step Functions or Airflow — which runs your pipelines? Tell me why in the comments.


Share this article

LinkedIn post draft