AWS Learning Journey

One Push to Production: A Pipeline That Tests, Scans, Deploys, and Rolls Back

A hands-on lab: build a GitHub Actions pipeline that tests the API, runs security scans, authenticates to AWS with OIDC, pushes to ECR, gates on vulnerabilities, deploys to ECS, smoke-tests — and rolls back automatically when something breaks.

14 min read · 2630 words
GitHub ActionsAWSCI/CDDevSecOpsTutorial
One Push to Production: A Pipeline That Tests, Scans, Deploys, and Rolls Back cover image

AWS from Zero to Production — Phase 7, Part 5 of 5 · See the full series roadmap


Everything comes together in this lab.

The OIDC trust from Phase 2. The network from Phase 3. The ECS service from Phase 4. The database and secret from Phase 5. The Terraform from Phase 6. And the CI/CD and security ideas from this phase.

By the end, git push will be the only thing you do to ship. The pipeline will build, test, scan, deploy, verify — and if the new version breaks, put the old one back without you lifting a finger.

What you’ll build

A single GitHub Actions workflow that:

  1. ✅ Builds and tests the API with pytest
  2. 🔍 Scans the code (Bandit), dependencies (pip-audit), and Terraform (Checkov)
  3. 🔐 Authenticates to AWS with OIDC — no stored keys
  4. 🐳 Pushes the image to ECR, tagged with the commit SHA
  5. 🚧 Gates on the ECR scan — critical vulnerabilities block the deploy
  6. 📝 Updates the ECS task definition
  7. 🚀 Deploys the ECS service and waits for it to be stable
  8. 🧪 Runs a smoke test against the load balancer
  9. ⏪ Rolls back automatically if anything after the deploy fails
push to main
   |
   +--> test (pytest) ----------+
   |                            |
   +--> scan (bandit, pip-audit,+--> deploy (environment: production)
        checkov)                     |  OIDC -> IAM role
                                     |  build + push to ECR
                                     |  wait for scan -> gate on CRITICAL
                                     |  render + deploy task definition
                                     |  smoke test
                                     |  on failure -> roll back

Prerequisites

Cost note: The deployed stack costs the same as in Phase 6 — NAT gateway, load balancer, Fargate, and RDS — until you destroy it. GitHub Actions minutes are free for public repositories and included in your plan for private ones.


Step 1: Create the repository

Create a new GitHub repository called orders-platform, and lay it out like this:

orders-platform/
  app/
    app.py
    requirements.txt
    requirements-dev.txt
    Dockerfile
    tests/
      test_app.py
  infra/
    (all the .tf files from the Phase 6 lab)
  .github/
    workflows/
      pipeline.yml
    dependabot.yml
  .gitignore

Copy the Phase 5 app into app/ and the Phase 6 Terraform files into infra/. Don’t copy .terraform/ or any state files. Use the .gitignore from Phase 6.


Step 2: Make the app testable

The Phase 5 app connects to the database and creates an AWS client the moment it’s imported — which makes unit tests impossible without AWS. Two small changes fix that.

In app/app.py, read the database settings with .get():

DB_HOST = os.environ.get("DB_HOST")
DB_NAME = os.environ.get("DB_NAME", "orders")
SECRET_ARN = os.environ.get("DB_SECRET_ARN")

Create the Secrets Manager client lazily, on first use:

_secrets = None
_creds = None


def get_creds(refresh=False):
    global _secrets, _creds
    if _secrets is None:
        _secrets = boto3.client("secretsmanager")
    if _creds is None or refresh:
        value = _secrets.get_secret_value(SecretId=SECRET_ARN)
        _creds = json.loads(value["SecretString"])
    return _creds

And only initialize the database when one is configured — replace the bare init_db() call:

if DB_HOST:
    init_db()

Create app/requirements-dev.txt:

pytest==8.*

Create app/tests/test_app.py:

from app import app


def test_health():
    response = app.test_client().get("/health")
    assert response.status_code == 200
    assert response.get_json() == {"status": "ok"}


def test_index_reports_version():
    response = app.test_client().get("/")
    assert response.status_code == 200
    assert "version" in response.get_json()


def test_create_order_rejects_missing_item():
    response = app.test_client().post("/orders", json={})
    assert response.status_code == 400


def test_create_order_rejects_long_item():
    response = app.test_client().post("/orders", json={"item": "x" * 201})
    assert response.status_code == 400

Run them locally:

cd app
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt
pytest -q
cd ..

Add .venv/ to .gitignore.

✅ Checkpoint: 4 passed.


Step 3: Hand image deployments to the pipeline

From now on, two tools touch the ECS service:

Without a rule, Terraform would see the pipeline’s new revisions as drift and try to undo them. Tell Terraform to leave that one attribute alone. In infra/ecs.tf, add to aws_ecs_service.app:

  lifecycle {
    ignore_changes = [task_definition]
  }

Why this is OK: This is a deliberate, documented ownership split — not a hidden manual change. Everything else about the service is still managed by Terraform.


Step 4: Add the deployment role to Terraform

Create infra/github-oidc.tf. It defines an IAM role that only your repository’s production environment can assume, with only the permissions the pipeline needs:

variable "github_owner" {
  type = string
}

variable "github_repo" {
  type    = string
  default = "orders-platform"
}

variable "github_owner_id" {
  type = string
}

variable "github_repo_id" {
  type = string
}

data "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"
}

data "aws_iam_policy_document" "github_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [data.aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values = [
        "repo:${var.github_owner}@${var.github_owner_id}/${var.github_repo}@${var.github_repo_id}:environment:production",
        "repo:${var.github_owner}/${var.github_repo}:environment:production",
      ]
    }
  }
}

resource "aws_iam_role" "github_deploy" {
  name               = "${local.prefix}-github-deploy"
  assume_role_policy = data.aws_iam_policy_document.github_trust.json
}

data "aws_iam_policy_document" "github_deploy" {
  statement {
    sid       = "EcrAuth"
    actions   = ["ecr:GetAuthorizationToken"]
    resources = ["*"]
  }

  statement {
    sid = "EcrPushAndScan"
    actions = [
      "ecr:BatchCheckLayerAvailability",
      "ecr:InitiateLayerUpload",
      "ecr:UploadLayerPart",
      "ecr:CompleteLayerUpload",
      "ecr:PutImage",
      "ecr:BatchGetImage",
      "ecr:DescribeImages",
      "ecr:DescribeImageScanFindings",
    ]
    resources = [aws_ecr_repository.app.arn]
  }

  statement {
    sid       = "TaskDefinitions"
    actions   = ["ecs:DescribeTaskDefinition", "ecs:RegisterTaskDefinition"]
    resources = ["*"]
  }

  statement {
    sid       = "Service"
    actions   = ["ecs:DescribeServices", "ecs:UpdateService"]
    resources = [aws_ecs_service.app.id]
  }

  statement {
    sid       = "PassTaskRoles"
    actions   = ["iam:PassRole"]
    resources = [aws_iam_role.execution.arn, aws_iam_role.task.arn]

    condition {
      test     = "StringEquals"
      variable = "iam:PassedToService"
      values   = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role_policy" "github_deploy" {
  name   = "deploy-orders"
  role   = aws_iam_role.github_deploy.id
  policy = data.aws_iam_policy_document.github_deploy.json
}

output "github_deploy_role_arn" {
  value = aws_iam_role.github_deploy.arn
}

Notice what the role can’t do: it can’t touch the database, the network, IAM policies, or any other repository or service. It can push one image repository and update one service.

Deleted the OIDC provider after Phase 2? Replace the data block with an aws_iam_openid_connect_provider resource with url = "https://token.actions.githubusercontent.com" and client_id_list = ["sts.amazonaws.com"], and update the reference.

Get your GitHub IDs — as in Phase 2, Step 3:

gh api repos/YOUR_GITHUB_USER/orders-platform --jq '{repo_id: .id, owner_id: .owner.id}'

Add them to infra/terraform.tfvars:

github_owner    = "YOUR_GITHUB_USER"
github_owner_id = "12345678"
github_repo_id  = "987654321"

Step 5: Apply the infrastructure

If you destroyed the Phase 6 stack, this rebuilds it. If it’s still running, this adds the role and the lifecycle rule.

cd infra
export AWS_PROFILE=sandbox
terraform init
terraform plan -out=tfplan
terraform apply tfplan

If the stack is brand new, push the first image exactly as in Phase 6, Step 9 — the pipeline needs a running service to update.

Collect the values the pipeline needs:

terraform output -raw github_deploy_role_arn
terraform output -raw alb_url
cd ..

✅ Checkpoint: The role orders-dev-github-deploy exists, and the app responds at the ALB URL.


Step 6: Configure GitHub

In the repository settings:

  1. Environments → New environment → production. Optional: add yourself as a required reviewer. Under Deployment branches and tags, choose Selected branches and allow only main.
  2. In the production environment, add three variables: AWS_ROLE_ARN (the role ARN from Step 5), AWS_REGION (us-east-1), and ALB_URL (the load balancer URL from Step 5).
  3. Advanced Security (or Code security): enable Dependabot alerts, secret scanning, and push protection, where available for your repository and plan.
  4. Branch protection or rulesets for main: require a pull request and require the test and scan checks to pass.

No secrets. Not one.


Step 7: Add Dependabot

Create .github/dependabot.yml, so dependencies, actions, and the base image stay current:

version: 2
updates:
  - package-ecosystem: pip
    directory: /app
    schedule:
      interval: weekly
  - package-ecosystem: docker
    directory: /app
    schedule:
      interval: weekly
  - package-ecosystem: github-actions
    directory: /
    schedule:
      interval: weekly

Step 8: Write the pipeline

Create .github/workflows/pipeline.yml:

name: pipeline

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

concurrency:
  group: pipeline-${{ github.ref }}
  cancel-in-progress: false

env:
  ECR_REPOSITORY: orders
  ECS_CLUSTER: orders-dev
  ECS_SERVICE: orders
  CONTAINER_NAME: orders

jobs:
  test:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: app
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
          cache: pip
          cache-dependency-path: app/requirements*.txt
      - run: pip install -r requirements.txt -r requirements-dev.txt
      - run: pytest -q

  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: pip install bandit pip-audit checkov
      - name: SAST (Bandit)
        run: bandit -r app -x app/tests -ll
      - name: Dependency audit (pip-audit)
        run: pip-audit -r app/requirements.txt
      - name: IaC scan (Checkov)
        run: checkov -d infra --framework terraform --quiet --compact --soft-fail

  deploy:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    needs: [test, scan]
    runs-on: ubuntu-latest
    environment: production
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v5

      - name: Authenticate to AWS (OIDC)
        uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}

      - name: Log in to Amazon ECR
        id: ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push the image
        id: image
        env:
          IMAGE: ${{ steps.ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}:${{ github.sha }}
        run: |
          docker build --platform linux/amd64 -t "$IMAGE" app
          docker push "$IMAGE"
          echo "image=$IMAGE" >> "$GITHUB_OUTPUT"

      - name: Security gate (ECR scan)
        env:
          TAG: ${{ github.sha }}
        run: |
          aws ecr wait image-scan-complete \
            --repository-name "$ECR_REPOSITORY" --image-id imageTag="$TAG"
          aws ecr describe-image-scan-findings \
            --repository-name "$ECR_REPOSITORY" --image-id imageTag="$TAG" \
            --query 'imageScanFindings.findingSeverityCounts' --output table || true
          CRITICAL=$(aws ecr describe-image-scan-findings \
            --repository-name "$ECR_REPOSITORY" --image-id imageTag="$TAG" \
            --query 'imageScanFindings.findingSeverityCounts.CRITICAL' --output text)
          if [ "$CRITICAL" != "None" ] && [ "$CRITICAL" -gt 0 ]; then
            echo "::error::Blocked: $CRITICAL critical vulnerabilities in the image"
            exit 1
          fi

      - name: Save the current task definition
        id: current
        run: |
          CURRENT=$(aws ecs describe-services \
            --cluster "$ECS_CLUSTER" --services "$ECS_SERVICE" \
            --query 'services[0].taskDefinition' --output text)
          echo "taskdef=$CURRENT" >> "$GITHUB_OUTPUT"
          aws ecs describe-task-definition --task-definition "$CURRENT" \
            --query taskDefinition > taskdef.json

      - name: Render the new task definition
        id: render
        uses: aws-actions/amazon-ecs-render-task-definition@v1
        with:
          task-definition: taskdef.json
          container-name: ${{ env.CONTAINER_NAME }}
          image: ${{ steps.image.outputs.image }}
          environment-variables: |
            APP_VERSION=${{ github.sha }}

      - name: Deploy to ECS
        uses: aws-actions/amazon-ecs-deploy-task-definition@v2
        with:
          task-definition: ${{ steps.render.outputs.task-definition }}
          cluster: ${{ env.ECS_CLUSTER }}
          service: ${{ env.ECS_SERVICE }}
          wait-for-service-stability: true
          wait-max-delay-seconds: 15

      - name: Smoke test
        env:
          ALB_URL: ${{ vars.ALB_URL }}
          EXPECTED: ${{ github.sha }}
        run: |
          for attempt in 1 2 3 4 5; do
            VERSION=$(curl -fsS "$ALB_URL/" | jq -r .version) || VERSION=""
            [ "$VERSION" = "$EXPECTED" ] && break
            echo "Waiting for the new version (got: $VERSION)"; sleep 6
          done
          [ "$VERSION" = "$EXPECTED" ] || { echo "::error::Expected $EXPECTED, got $VERSION"; exit 1; }
          curl -fsS "$ALB_URL/health"
          curl -fsS "$ALB_URL/orders" > /dev/null
          echo "Smoke test passed for $EXPECTED"

      - name: Roll back on failure
        if: failure() && steps.current.outputs.taskdef != ''
        env:
          PREVIOUS: ${{ steps.current.outputs.taskdef }}
        run: |
          echo "::warning::Rolling back to $PREVIOUS"
          aws ecs update-service --cluster "$ECS_CLUSTER" --service "$ECS_SERVICE" \
            --task-definition "$PREVIOUS" > /dev/null
          aws ecs wait services-stable --cluster "$ECS_CLUSTER" --services "$ECS_SERVICE"
          echo "Rolled back to $PREVIOUS"

How it works

Pin by SHA in production. The workflow uses major version tags for readability. In real pipelines, pin third-party actions to full commit SHAs, and let Dependabot update them.


Step 9: Ship it

git add .
git commit -m "Add orders platform with CI/CD pipeline"
git push origin main

Open the Actions tab. Watch test and scan run in parallel, then deploy — waiting for your approval if you added a reviewer.

When it finishes:

curl -s "$(cd infra && terraform output -raw alb_url)/"

✅ Checkpoint: The version field is the full SHA of your commit. The ECS service is on a new task definition revision registered by the pipeline.


Step 10: Break it on purpose — and watch it roll back

Introduce a bug that passes the unit tests and passes the health check, but breaks a real endpoint. In app/app.py, misspell the table name in list_orders:

            "SELECT id, item, created_at FROM orderz ORDER BY id DESC LIMIT 20"

Commit and push:

git commit -am "Refactor orders query"
git push origin main

Watch the pipeline:

  1. Tests pass — they don’t hit the database.
  2. The deploy succeeds — /health is fine, so the tasks are healthy.
  3. The smoke test fails — GET /orders returns a 500.
  4. The rollback step runs and puts the previous task definition back.

✅ Checkpoint: The workflow run is red, the warning shows Rolling back to ..., and curl $ALB_URL/ reports the previous commit SHA. Production was broken for well under a couple of minutes, with no human involved.

Fix the typo, push again, and watch it go green.

Lesson: Health checks tell you the app is running. Smoke tests tell you it’s working. You need both.


Step 11 (optional): Trip the dependency gate

Add a known-vulnerable package to app/requirements.txt in a branch:

requests==2.19.0

Open a pull request. The scan job fails at pip-audit, listing the CVEs — and branch protection stops the merge.

Close the pull request without merging.


Cleanup

cd infra
terraform destroy

This removes the whole stack, including the deployment role. Then, in GitHub, delete the orders-platform repository or keep it as a reference.

Keep the GitHub OIDC provider if you want to keep experimenting — it has no additional charge. Delete the Terraform state bucket as described in Phase 6 if you’re finished with it.

✅ Final checkpoint: terraform destroy reports Destroy complete!


Troubleshooting

Not authorized to perform sts:AssumeRoleWithWebIdentity Cause: the sub doesn’t match — wrong owner or repository name, wrong IDs, or the job isn’t using environment: production. Fix: check CloudTrail’s AssumeRoleWithWebIdentity event for the actual sub, as in Phase 2.

AccessDeniedException on iam:PassRole Cause: the task definition references a role the policy doesn’t list. Fix: make sure the task definition uses the Terraform-created execution and task roles.

aws ecr wait image-scan-complete times out Cause: scan-on-push is off, or the repository uses enhanced scanning, which reports differently. Fix: check the repository’s scan settings.

Terraform wants to change the task definition back Cause: the ignore_changes lifecycle rule is missing. Fix: add it as in Step 3.

The deploy job never starts Cause: it’s waiting for a required reviewer, or the push wasn’t to main. Fix: approve the deployment, or check the branch.

The smoke test fails on the first deploy after a fresh terraform apply Cause: the database table doesn’t exist yet, or tasks are still starting. Fix: wait a minute and re-run the job.


What you accomplished

This is the capstone’s delivery path: GitHub Actions → OIDC → IAM role → ECR → ECS.


Quick self-check

Test yourself before moving on:

  1. Why can’t a pull request from a fork deploy to production?
  2. Why does Terraform ignore changes to the service’s task definition?
  3. Which failure does the circuit breaker catch, and which does the smoke test catch?
  4. Why is --soft-fail on Checkov acceptable for now, and what should replace it?

Answers

  1. The deploy job only runs on pushes to main, and only the production environment’s OIDC sub can assume the role.
  2. The pipeline owns image rollouts; without the rule, Terraform would revert every deployment.
  3. The circuit breaker catches tasks that never become healthy; the smoke test catches versions that run but don’t work.
  4. The lab infrastructure intentionally skips some production settings. Replace it with fixes and documented suppressions, then remove --soft-fail.

What’s next

Phase 8 is about running what you’ve built: CloudWatch metrics, logs, alarms, and dashboards; auditing and troubleshooting; reliability engineering; and the AWS Well-Architected Framework. Its lab builds a dashboard for this exact service.

How long did your automatic rollback take? Post your numbers in the comments.


Share this article

LinkedIn post draft