AWS Learning Journey

Rebuild Everything With One Command: Your AWS Stack in Terraform

A hands-on lab: provision the VPC, ECR, ECS on Fargate, RDS, IAM roles, and CloudWatch alarms with Terraform — using remote state in S3 with locking — then catch a manual change with a plan.

12 min read · 2264 words
TerraformAWSInfrastructure As CodeTutorialDevOps
Rebuild Everything With One Command: Your AWS Stack in Terraform cover image

AWS from Zero to Production — Phase 6, Part 5 of 5 · See the full series roadmap


In Phases 3 through 5, you built a production-shaped stack by hand: a two-AZ VPC, a load balancer, an ECS service on Fargate, a private PostgreSQL database with a managed secret, and the IAM roles to tie it together.

It took a few hours. It took dozens of commands. And if you had to do it again tomorrow — or for a second environment — you’d have to do it all again, and hope you didn’t miss a step.

In this lab, you’ll write it once, as Terraform. Then terraform apply builds the whole thing, terraform destroy removes it, and a pull request is all it takes to change it.

What you’ll build

orders-infra/
  versions.tf      Terraform, provider, backend
  variables.tf     inputs
  network.tf       VPC module
  security.tf      security groups
  ecr.tf           repository + lifecycle policy
  iam.tf           execution and task roles
  database.tf      RDS PostgreSQL
  alb.tf           load balancer, target group, listener
  ecs.tf           cluster, log group, task definition, service
  monitoring.tf    SNS topic and CloudWatch alarms
  outputs.tf       useful values

Prerequisites

Cost note: This creates the same billable resources as Phases 4 and 5 — NAT gateway, load balancer, Fargate tasks, RDS, and public IPv4 addresses — plus a tiny S3 state bucket. Run terraform destroy when you’re done. Check current pricing before you start.


Step 1: Create the state bucket

Terraform can’t store its state in a bucket it hasn’t created yet, so the state bucket is made once, by hand. This is the only resource you’ll create outside Terraform:

export AWS_PROFILE=sandbox
export AWS_REGION=us-east-1
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
STATE_BUCKET="tfstate-$ACCOUNT_ID-$AWS_REGION"

aws s3api create-bucket --bucket "$STATE_BUCKET"

aws s3api put-bucket-versioning --bucket "$STATE_BUCKET" \
  --versioning-configuration Status=Enabled

aws s3api put-public-access-block --bucket "$STATE_BUCKET" \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

echo "$STATE_BUCKET"

New buckets are encrypted by default. Versioning lets you recover an older state if something goes wrong.

Outside us-east-1? Add --create-bucket-configuration LocationConstraint=$AWS_REGION to create-bucket.

✅ Checkpoint: The bucket exists, with versioning enabled and public access blocked.


Step 2: Set up the project

mkdir orders-infra && cd orders-infra

versions.tf

Replace REPLACE_WITH_STATE_BUCKET with the bucket name from Step 1. Backend settings can’t use variables.

terraform {
  required_version = ">= 1.11"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }

  backend "s3" {
    bucket       = "REPLACE_WITH_STATE_BUCKET"
    key          = "orders/dev/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

provider "aws" {
  region = var.region

  default_tags {
    tags = {
      Project     = var.name
      Environment = var.environment
      ManagedBy   = "terraform"
    }
  }
}

variables.tf

variable "region" {
  type    = string
  default = "us-east-1"
}

variable "name" {
  type    = string
  default = "orders"
}

variable "environment" {
  type    = string
  default = "dev"
}

variable "image_tag" {
  type        = string
  description = "Container image tag to deploy"
}

variable "desired_count" {
  type    = number
  default = 2
}

variable "alert_email" {
  type        = string
  description = "Email address for alarm notifications"
}

locals {
  prefix = "${var.name}-${var.environment}"
}

terraform.tfvars

image_tag   = "3.0.0"
alert_email = "you@example.com"

Add a .gitignore before you ever commit:

.terraform/
*.tfstate
*.tfstate.*
*.tfplan

Step 3: The network

network.tf

data "aws_availability_zones" "available" {
  state = "available"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"

  name = local.prefix
  cidr = "10.0.0.0/16"
  azs  = slice(data.aws_availability_zones.available.names, 0, 2)

  public_subnets   = ["10.0.1.0/24", "10.0.2.0/24"]
  private_subnets  = ["10.0.11.0/24", "10.0.12.0/24"]
  database_subnets = ["10.0.21.0/24", "10.0.22.0/24"]

  create_database_subnet_group = true

  enable_nat_gateway = true
  single_nat_gateway = true
}

This one module creates the VPC, six subnets across two AZs, an internet gateway, a NAT gateway, route tables, and a DB subnet group — the same three-tier layout you built by hand in Phase 3.

security.tf

resource "aws_security_group" "alb" {
  name        = "${local.prefix}-alb"
  description = "Load balancer"
  vpc_id      = module.vpc.vpc_id
}

resource "aws_security_group" "app" {
  name        = "${local.prefix}-app"
  description = "ECS tasks"
  vpc_id      = module.vpc.vpc_id
}

resource "aws_security_group" "db" {
  name        = "${local.prefix}-db"
  description = "PostgreSQL"
  vpc_id      = module.vpc.vpc_id
}

resource "aws_vpc_security_group_ingress_rule" "alb_http" {
  security_group_id = aws_security_group.alb.id
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "tcp"
  from_port         = 80
  to_port           = 80
}

resource "aws_vpc_security_group_egress_rule" "alb_to_app" {
  security_group_id            = aws_security_group.alb.id
  referenced_security_group_id = aws_security_group.app.id
  ip_protocol                  = "tcp"
  from_port                    = 8080
  to_port                      = 8080
}

resource "aws_vpc_security_group_ingress_rule" "app_from_alb" {
  security_group_id            = aws_security_group.app.id
  referenced_security_group_id = aws_security_group.alb.id
  ip_protocol                  = "tcp"
  from_port                    = 8080
  to_port                      = 8080
}

resource "aws_vpc_security_group_egress_rule" "app_all" {
  security_group_id = aws_security_group.app.id
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "-1"
}

resource "aws_vpc_security_group_ingress_rule" "db_from_app" {
  security_group_id            = aws_security_group.db.id
  referenced_security_group_id = aws_security_group.app.id
  ip_protocol                  = "tcp"
  from_port                    = 5432
  to_port                      = 5432
}

Terraform gotcha: Unlike the console, a security group created by Terraform has no outbound rules until you add them. That’s why the load balancer and app get explicit egress rules. The database needs none.


Step 4: ECR and IAM

ecr.tf

resource "aws_ecr_repository" "app" {
  name                 = var.name
  image_tag_mutability = "IMMUTABLE"
  force_delete         = true # lab only: lets destroy remove images

  image_scanning_configuration {
    scan_on_push = true
  }
}

resource "aws_ecr_lifecycle_policy" "app" {
  repository = aws_ecr_repository.app.name

  policy = jsonencode({
    rules = [{
      rulePriority = 1
      description  = "Keep the 10 most recent images"
      selection = {
        tagStatus   = "any"
        countType   = "imageCountMoreThan"
        countNumber = 10
      }
      action = { type = "expire" }
    }]
  })
}

iam.tf

data "aws_iam_policy_document" "ecs_tasks_assume" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "execution" {
  name               = "${local.prefix}-ecs-execution"
  assume_role_policy = data.aws_iam_policy_document.ecs_tasks_assume.json
}

resource "aws_iam_role_policy_attachment" "execution" {
  role       = aws_iam_role.execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_iam_role" "task" {
  name               = "${local.prefix}-task"
  assume_role_policy = data.aws_iam_policy_document.ecs_tasks_assume.json
}

resource "aws_iam_role_policy" "task_read_db_secret" {
  name = "read-db-secret"
  role = aws_iam_role.task.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = "secretsmanager:GetSecretValue"
      Resource = aws_db_instance.main.master_user_secret[0].secret_arn
    }]
  })
}

The task role’s policy references the database’s secret ARN directly — no copying and pasting, and it can never point at the wrong secret.


Step 5: The database

database.tf

resource "aws_db_instance" "main" {
  identifier     = "${local.prefix}-db"
  engine         = "postgres"
  instance_class = "db.t4g.micro"

  allocated_storage = 20
  storage_type      = "gp3"
  storage_encrypted = true

  db_name                     = "orders"
  username                    = "orders_admin"
  manage_master_user_password = true

  db_subnet_group_name   = module.vpc.database_subnet_group_name
  vpc_security_group_ids = [aws_security_group.db.id]
  publicly_accessible    = false
  multi_az               = false # use true in production

  backup_retention_period = 7

  deletion_protection = false # lab only
  skip_final_snapshot = true  # lab only
}

No password appears anywhere — in code, in variables, or in Terraform state. RDS creates it in Secrets Manager.

In production: set multi_az = true, deletion_protection = true, skip_final_snapshot = false, and add lifecycle { prevent_destroy = true }.


Step 6: The load balancer and ECS service

alb.tf

resource "aws_lb" "main" {
  name               = "${local.prefix}-alb"
  load_balancer_type = "application"
  subnets            = module.vpc.public_subnets
  security_groups    = [aws_security_group.alb.id]
}

resource "aws_lb_target_group" "app" {
  name                 = "${local.prefix}-tg"
  port                 = 8080
  protocol             = "HTTP"
  vpc_id               = module.vpc.vpc_id
  target_type          = "ip"
  deregistration_delay = 30

  health_check {
    path                = "/health"
    interval            = 15
    healthy_threshold   = 2
    unhealthy_threshold = 2
  }
}

resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.main.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}

ecs.tf

resource "aws_ecs_cluster" "main" {
  name = local.prefix
}

resource "aws_cloudwatch_log_group" "app" {
  name              = "/ecs/${local.prefix}"
  retention_in_days = 14
}

resource "aws_ecs_task_definition" "app" {
  family                   = local.prefix
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 256
  memory                   = 512
  execution_role_arn       = aws_iam_role.execution.arn
  task_role_arn            = aws_iam_role.task.arn

  runtime_platform {
    cpu_architecture        = "X86_64"
    operating_system_family = "LINUX"
  }

  container_definitions = jsonencode([{
    name         = var.name
    image        = "${aws_ecr_repository.app.repository_url}:${var.image_tag}"
    essential    = true
    portMappings = [{ containerPort = 8080, protocol = "tcp" }]
    environment = [
      { name = "APP_VERSION", value = var.image_tag },
      { name = "DB_HOST", value = aws_db_instance.main.address },
      { name = "DB_NAME", value = aws_db_instance.main.db_name },
      { name = "DB_SECRET_ARN", value = aws_db_instance.main.master_user_secret[0].secret_arn },
    ]
    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = aws_cloudwatch_log_group.app.name
        "awslogs-region"        = var.region
        "awslogs-stream-prefix" = "app"
      }
    }
  }])
}

resource "aws_ecs_service" "app" {
  name            = var.name
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = var.desired_count
  launch_type     = "FARGATE"

  health_check_grace_period_seconds  = 30
  deployment_minimum_healthy_percent = 100
  deployment_maximum_percent         = 200

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  network_configuration {
    subnets          = module.vpc.private_subnets
    security_groups  = [aws_security_group.app.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.app.arn
    container_name   = var.name
    container_port   = 8080
  }

  depends_on = [aws_lb_listener.http]
}

Every value that used to be a shell variable — the image URL, the database address, the secret ARN, the subnet IDs — is now a reference. Terraform works out the order and fills them in.


Step 7: Monitoring and outputs

monitoring.tf

resource "aws_sns_topic" "alerts" {
  name = "${local.prefix}-alerts"
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

resource "aws_cloudwatch_metric_alarm" "alb_5xx" {
  alarm_name          = "${local.prefix}-target-5xx"
  alarm_description   = "The app is returning 5xx errors"
  namespace           = "AWS/ApplicationELB"
  metric_name         = "HTTPCode_Target_5XX_Count"
  dimensions          = { LoadBalancer = aws_lb.main.arn_suffix }
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 1
  threshold           = 10
  comparison_operator = "GreaterThanThreshold"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [aws_sns_topic.alerts.arn]
  ok_actions          = [aws_sns_topic.alerts.arn]
}

resource "aws_cloudwatch_metric_alarm" "unhealthy_targets" {
  alarm_name          = "${local.prefix}-unhealthy-targets"
  alarm_description   = "At least one target is failing health checks"
  namespace           = "AWS/ApplicationELB"
  metric_name         = "UnHealthyHostCount"
  dimensions = {
    LoadBalancer = aws_lb.main.arn_suffix
    TargetGroup  = aws_lb_target_group.app.arn_suffix
  }
  statistic           = "Maximum"
  period              = 60
  evaluation_periods  = 3
  threshold           = 0
  comparison_operator = "GreaterThanThreshold"
  alarm_actions       = [aws_sns_topic.alerts.arn]
  ok_actions          = [aws_sns_topic.alerts.arn]
}

resource "aws_cloudwatch_metric_alarm" "db_cpu" {
  alarm_name          = "${local.prefix}-db-cpu"
  alarm_description   = "Database CPU is high"
  namespace           = "AWS/RDS"
  metric_name         = "CPUUtilization"
  dimensions          = { DBInstanceIdentifier = aws_db_instance.main.identifier }
  statistic           = "Average"
  period              = 300
  evaluation_periods  = 3
  threshold           = 80
  comparison_operator = "GreaterThanThreshold"
  alarm_actions       = [aws_sns_topic.alerts.arn]
}

resource "aws_cloudwatch_metric_alarm" "db_storage" {
  alarm_name          = "${local.prefix}-db-free-storage"
  alarm_description   = "Database free storage is below 2 GiB"
  namespace           = "AWS/RDS"
  metric_name         = "FreeStorageSpace"
  dimensions          = { DBInstanceIdentifier = aws_db_instance.main.identifier }
  statistic           = "Minimum"
  period              = 300
  evaluation_periods  = 1
  threshold           = 2147483648
  comparison_operator = "LessThanThreshold"
  alarm_actions       = [aws_sns_topic.alerts.arn]
}

outputs.tf

output "alb_url" {
  value = "http://${aws_lb.main.dns_name}"
}

output "ecr_repository_url" {
  value = aws_ecr_repository.app.repository_url
}

output "db_endpoint" {
  value = aws_db_instance.main.address
}

Step 8: Initialize and validate

terraform init
terraform fmt
terraform validate

init downloads the AWS provider and VPC module, and connects to your S3 backend.

✅ Checkpoint: terraform validate prints Success! The configuration is valid. Commit .terraform.lock.hcl along with your code.


Step 9: Create the repository and push the image

The ECS service needs an image to run, and the image needs a repository. Create just the repository first:

terraform apply -target=aws_ecr_repository.app

About -target: It’s meant for exceptional cases like this bootstrap step — not everyday use. Many teams keep ECR repositories in a separate, long-lived configuration instead.

Then build and push, from your orders-api folder:

REPO_URL=$(terraform output -raw ecr_repository_url)
REGISTRY="${REPO_URL%%/*}"

aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY"

docker build --platform linux/amd64 -t "$REPO_URL:3.0.0" ../orders-api
docker push "$REPO_URL:3.0.0"

Adjust ../orders-api to wherever your app folder is.

✅ Checkpoint: The image orders:3.0.0 is in the ECR repository.


Step 10: Plan and apply everything

terraform plan -out=tfplan

Read the plan. You’ll see around 50 resources to create — and nothing to change or destroy. Then:

terraform apply tfplan

This takes 10–15 minutes, mostly waiting for the database and NAT gateway.

Check your email and confirm the SNS subscription, or alarms can’t reach you.

✅ Checkpoint: The apply finishes with Apply complete! and prints alb_url.


Step 11: Test the app

ALB_URL=$(terraform output -raw alb_url)

curl -s -X POST "$ALB_URL/orders" \
  -H "Content-Type: application/json" -d '{"item": "built by terraform"}'

curl -s "$ALB_URL/orders"

It may take a minute after apply for the tasks to become healthy.

✅ Checkpoint: The same orders API from Phase 5 — but this time, every piece of infrastructure came from code.


Step 12: Catch a manual change

Simulate a colleague “quickly fixing” something in the console:

aws ecs update-service --cluster orders-dev --service orders --desired-count 3

Now ask Terraform what it thinks:

terraform plan

The plan shows desired_count changing from 3 back to 2. Terraform noticed the drift and wants to restore what the code says.

You have two choices — and this is the real lesson of IaC:

Either way, the code stays the source of truth.

terraform apply

✅ Checkpoint: The service is back to 2 tasks, and terraform plan shows No changes.


Step 13: Ship a change through code

Change something real: in variables.tf, set the desired_count default to 3. Then:

terraform plan -out=tfplan
terraform apply tfplan

The plan shows exactly one in-place update. That’s what infrastructure changes should look like: small, reviewed, and predictable.


Cleanup

terraform destroy

Review the plan — every resource listed for destruction — and type yes. It takes 10–15 minutes.

Then remove the state bucket. Because it’s versioned, empty it first in the S3 console (Empty, then Delete), or with:

aws s3api delete-objects --bucket "$STATE_BUCKET" \
  --delete "$(aws s3api list-object-versions --bucket "$STATE_BUCKET" \
    --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' --output json)"
aws s3api delete-objects --bucket "$STATE_BUCKET" \
  --delete "$(aws s3api list-object-versions --bucket "$STATE_BUCKET" \
    --query '{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}' --output json)"
aws s3api delete-bucket --bucket "$STATE_BUCKET"

Keep it instead? If you’re continuing to Phase 7, you can keep the state bucket — it costs almost nothing — and reuse this configuration for the CI/CD lab.

✅ Final checkpoint: terraform destroy reports Destroy complete!, and there are no NAT gateways, load balancers, or RDS instances left.


Troubleshooting

Error acquiring the state lock Cause: another run is in progress, or a previous run crashed. Fix: wait, or — if you’re sure nothing else is running — terraform force-unlock LOCK_ID.

Failed to get existing workspaces: S3 bucket does not exist Cause: the backend bucket name is wrong. Fix: check versions.tf and rerun terraform init.

ECS deployment fails, and the circuit breaker rolls back Cause: the image tag in terraform.tfvars doesn’t exist in ECR. Fix: push the image, or correct the tag.

Unsupported argument errors Cause: provider or module major version differences. Fix: check the pinned versions and the provider’s upgrade guide.

DependencyViolation during destroy Cause: network interfaces from the load balancer or tasks are still being released. Fix: run terraform destroy again after a few minutes.


What you accomplished


Quick self-check

Test yourself before moving on:

  1. Why is the state bucket created outside Terraform?
  2. Why do the app and load balancer security groups need explicit egress rules?
  3. How did the task role get the correct secret ARN?
  4. Someone changed the service in the console. How did you find out?

Answers

  1. Terraform needs the bucket to exist before it can store state in it.
  2. Terraform-created security groups have no outbound rules by default.
  3. By referencing aws_db_instance.main.master_user_secret[0].secret_arn.
  4. terraform plan showed the difference between the code and reality.

What’s next

Phase 7 automates everything you’ve done by hand so far. GitHub Actions will build and test the API, scan it, authenticate to AWS with OIDC, push to ECR, deploy to ECS, run a smoke test, and roll back automatically on failure.

How many resources did your plan create? And did drift detection surprise you? Tell me in the comments.


Share this article

LinkedIn post draft