AWS Learning Journey

It's Broken. Now What? Auditing and Troubleshooting on AWS

CloudTrail, CloudTrail Lake, AWS Config, VPC Flow Logs, load balancer and S3 access logs, RDS logs, ECS service events, Systems Manager, the AWS Health Dashboard, and Support — plus a playbook for the errors you'll actually see.

9 min read · 1726 words
AWSTroubleshootingDevOpsSite Reliability EngineeringCloud Security
It's Broken. Now What? Auditing and Troubleshooting on AWS cover image

AWS from Zero to Production — Phase 8, Part 2 of 5 · See the full series roadmap


It’s 2:14 a.m. The alarm says the orders API is returning 503s.

You could start clicking around the console at random. Or you could follow the evidence: the load balancer says there are no healthy targets. ECS service events say tasks are stopping. The stopped reason says CannotPullContainerError. CloudTrail shows someone changed the NAT gateway route an hour ago.

Fifteen minutes from page to root cause — because you knew where AWS keeps the evidence.

This article is a map of that evidence, and a playbook for using it.

What you’ll walk away with


The evidence map

Question                               Where to look
-------------------------------------  -----------------------------------------
Who changed something?                 CloudTrail
What did this resource look like?      AWS Config
Is traffic reaching the resource?      VPC Flow Logs
What did the load balancer see?        ELB access logs, ALB metrics
Who read or wrote this S3 object?      CloudTrail data events, S3 access logs
Why is the database slow?              RDS logs, Database Insights
Why are tasks stopping?                ECS service events, stopped reason
Is AWS itself having a problem?        AWS Health Dashboard
What is my app saying?                 CloudWatch Logs (Part 1)

AWS CloudTrail: who did what

CloudTrail records API calls made in your account — from the console, CLI, SDKs, and other AWS services.

Each event shows:

Event history vs. trails

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteRoute \
  --start-time 2026-10-31T00:00:00Z \
  --query 'Events[].[EventTime,Username,EventName]' --output table

CloudTrail Lake

CloudTrail Lake stores events in an event data store you can query with SQL, with long retention options:

SELECT eventTime, userIdentity.arn, eventName, sourceIPAddress
FROM <event-data-store-id>
WHERE eventSource = 'ec2.amazonaws.com'
  AND eventName LIKE 'Delete%'
  AND eventTime > '2026-10-30 00:00:00'
ORDER BY eventTime DESC

It’s the fastest way to answer “who deleted things across our accounts last night?” — without building your own pipeline.


AWS Config: what changed

Phase 2 introduced AWS Config for compliance. For troubleshooting, its superpower is the configuration timeline:

Pattern: “It worked yesterday” → open the resource in Config → compare yesterday’s configuration with today’s → follow the link to CloudTrail to see who changed it.


Network evidence

VPC Flow Logs

From Phase 3: flow logs record accepted and rejected traffic per network interface. If a connection times out, look for REJECT records:

fields @timestamp, srcAddr, dstAddr, dstPort, action
| filter dstPort = 5432 and action = "REJECT"
| sort @timestamp desc
| limit 20

A REJECT means a security group or network ACL blocked it. No record at all often means a routing problem — the traffic never got there.

Elastic Load Balancing access logs

ALB access logs record every request: client IP, target, request path, status codes from the load balancer and the target, and timing for each stage.


S3 access evidence

Two options, often confused:


RDS logs and database insights

For PostgreSQL, log slow queries by setting log_min_duration_statement in your parameter group — for example, to 500 ms.


ECS service events and stopped tasks

When an ECS service misbehaves, two places tell you why.

Service events

The service’s Events tab, or:

aws ecs describe-services --cluster orders-dev --services orders \
  --query 'services[0].events[:10].[createdAt,message]' --output table

Messages like:

Stopped task reasons

aws ecs list-tasks --cluster orders-dev --desired-status STOPPED \
  --query 'taskArns[:5]' --output text \
  | xargs aws ecs describe-tasks --cluster orders-dev \
      --query 'tasks[].[stoppedReason,containers[0].reason,containers[0].exitCode]' \
      --output table --tasks

Common answers:

Stopped tasks are only visible for a short time, so check promptly — or capture ECS events with EventBridge.


Systems Manager for troubleshooting

From Phase 6, Part 4:


AWS Health Dashboard

Sometimes it is AWS.

Check it early. Before spending an hour debugging, spend 30 seconds confirming the problem isn’t an AWS event in your Region.


Support Center

When you need AWS’s help:


A troubleshooting playbook

The ALB returns 502

The target closed the connection or sent an invalid response. Check app logs for crashes, and make sure the app’s keep-alive timeout is longer than the load balancer’s idle timeout.

The ALB returns 503

No healthy targets. Check target health, ECS service events, and stopped task reasons.

The ALB returns 504

The target took too long. Check app latency, slow database queries, and calls to downstream dependencies.

The connection times out

Something blocks or misroutes traffic. Walk the path from Phase 3: DNS, route tables, security groups, NACLs. Check flow logs for REJECT.

AccessDenied

Read the full error message — it usually names the action, the resource, and the type of policy that denied it. Check the identity’s policies, resource policies, permissions boundaries, SCPs, and KMS key policies. CloudTrail shows the exact request.

“It worked yesterday”

AWS Config timeline for the resources involved, then CloudTrail for the change, then your deployment history.

General method

  1. Scope: who’s affected, since when, and how badly?
  2. Check the obvious: recent deployments, recent changes, the Health Dashboard.
  3. Follow the request path, hop by hop, using the evidence map.
  4. Mitigate first — roll back, fail over, scale up — then find the root cause.
  5. Write it down for the postmortem (Part 3).

Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. An ALB returns 503 for every request. What’s the most likely cause?
  2. An ECS task stops with exit code 137. What happened?
  3. A security group changed overnight. How do you find out who changed it?
  4. How do you tell whether traffic was blocked or never arrived?

Answers

  1. No healthy targets in the target group.
  2. The container was killed for exceeding its memory limit.
  3. AWS Config timeline, linked to the CloudTrail event.
  4. VPC Flow Logs — REJECT means blocked; no record suggests a routing issue.

What’s next

In Part 3, we design for failure: RTO and RPO, disaster recovery strategies, multi-AZ and multi-Region patterns, fault injection, AWS Resilience Hub, DR testing, runbooks, and incident management.

What’s the hardest AWS issue you’ve ever debugged? Tell the story in the comments.


Share this article

LinkedIn post draft