AWS Learning Journey

If You Can't See It, You Can't Fix It: Amazon CloudWatch Explained

Metrics, logs, metric filters, Logs Insights, alarms, composite alarms, anomaly detection, dashboards, the CloudWatch agent, and Container Insights — observability on AWS, from first alarm to full picture.

8 min read · 1590 words
AWSObservabilityMonitoringDevOpsSite Reliability Engineering
If You Can't See It, You Can't Fix It: Amazon CloudWatch Explained cover image

AWS from Zero to Production — Phase 8, Part 1 of 5 · See the full series roadmap


How did you find out about your last production incident?

If the answer is “a customer told us” or “someone noticed on Twitter,” you don’t have a monitoring problem. You have a visibility problem. Your system was telling you something was wrong — in metrics and logs — and nobody was listening.

Amazon CloudWatch is how AWS listens. Every AWS service sends it metrics. Your apps send it logs. And CloudWatch can wake you up, page your team, or trigger automation the moment something drifts.

This article covers the whole toolkit — and how to use it without drowning in noise or cost.

What you’ll walk away with


The three signals

Observability usually means three kinds of data:

This article focuses on metrics and logs — the foundation everything else builds on.


Metrics

A metric is a time-ordered set of data points.

Choose the right statistic

Engineer’s note: If your average latency is 120 ms but your p99 is 4 seconds, one in a hundred requests is painfully slow. On a busy service, that’s thousands of unhappy users an hour.

Resolution and retention

Custom metrics

Publish your own business and application metrics:

aws cloudwatch put-metric-data \
  --namespace Orders/App \
  --metric-name OrdersCreated \
  --dimensions Environment=prod \
  --value 1

Or, more efficiently, write Embedded Metric Format (EMF) — structured JSON logs that CloudWatch turns into metrics automatically, with no extra API calls.

Cost tip: Each unique combination of metric name and dimensions is a separate custom metric, billed monthly. Never use high-cardinality values — like user IDs or request IDs — as dimensions.


Logs

Retention

By default, logs are kept forever. Set retention on every log group:

aws logs put-retention-policy --log-group-name /ecs/orders-dev --retention-in-days 30

For long-term or compliance storage, export to S3, or use the Infrequent Access log class for logs you rarely query.

Write structured logs

Logs that are JSON are far easier to query:

{"level":"error","msg":"payment failed","order_id":"A17","duration_ms":812,"status":502}

Than this:

ERROR payment failed for order A17 after 812ms (502)

Metric filters

A metric filter turns log patterns into metrics:

aws logs put-metric-filter \
  --log-group-name /ecs/orders-dev \
  --filter-name app-exceptions \
  --filter-pattern '"Exception on"' \
  --metric-transformations \
      metricName=AppExceptions,metricNamespace=Orders/App,metricValue=1,defaultValue=0

Now you can alarm on exceptions in your logs, graph them, and put them on dashboards. For JSON logs, filter patterns can match fields: { $.level = "error" }.

Subscription filters go further, streaming matching log events in real time to Lambda, Kinesis, Amazon Data Firehose, or OpenSearch.


CloudWatch Logs Insights

Logs Insights is an interactive query language for your logs:

fields @timestamp, @message
| filter @message like /Exception|Traceback/
| stats count(*) as errors by bin(5m)
| sort errors desc

For JSON logs, fields are discovered automatically:

fields @timestamp, order_id, duration_ms
| filter level = "error"
| stats avg(duration_ms), pct(duration_ms, 99) by status

Alarms

A CloudWatch alarm watches a metric and changes state when it crosses a threshold:

The settings that matter

Metric:              HTTPCode_Target_5XX_Count (Sum)
Period:              60 seconds
Threshold:           > 10
Datapoints to alarm: 3 out of 5
Missing data:        treat as not breaching
Actions:             SNS topic -> email, chat, or paging

Alert on symptoms, not causes

Page humans for things users feel: errors, latency, and availability. CPU at 85% isn’t an emergency if users are happy. Send cause-level alarms to dashboards or tickets instead.


Composite alarms

A composite alarm combines other alarms with AND, OR, and NOT:

ALARM("orders-target-5xx") AND ALARM("orders-db-cpu")

Use composite alarms to:


Anomaly detection

Some metrics don’t have a good fixed threshold. Traffic at 3 a.m. is very different from traffic at noon.

Anomaly detection learns a metric’s normal pattern — including daily and weekly cycles — and draws an expected band. Alarm when the metric leaves the band:

Alarm when RequestCount is outside the band (2 standard deviations)

Great for traffic drops (“why did orders fall by half?”), which fixed thresholds often miss.


Dashboards

A dashboard puts the metrics that matter on one screen.

A good service dashboard answers four questions at a glance — often called the golden signals:

Tips:


Collecting more data

The CloudWatch agent

EC2 instances don’t send memory or disk usage to CloudWatch by default — the hypervisor can’t see inside the OS. The CloudWatch agent collects:

Install and configure it with Systems Manager (Phase 6, Part 4), and store its configuration in Parameter Store.

Container Insights

Container Insights collects metrics and logs for ECS and EKS: CPU, memory, network, and storage per cluster, service, task, and container — plus running task counts. Enable it per cluster. Enhanced observability adds deeper, container-level detail.

Application Insights and Application Signals


Controlling CloudWatch cost


Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. Which statistic should you use for latency?
  2. Your EC2 dashboard doesn’t show memory usage. Why?
  3. How do you alarm on the number of exceptions written to logs?
  4. How do you avoid five separate pages for one incident?

Answers

  1. Percentiles — like p95 and p99.
  2. Memory isn’t visible to the hypervisor. Install the CloudWatch agent.
  3. Create a metric filter, then an alarm on the resulting metric.
  4. Use a composite alarm, and send notifications only from it.

What’s next

In Part 2, we go detective: CloudTrail and CloudTrail Lake, AWS Config, flow logs, load balancer and S3 access logs, RDS logs, ECS service events, the AWS Health Dashboard, and a practical troubleshooting playbook.

What’s the one alarm that’s saved your team the most? Share it in the comments.


Share this article

LinkedIn post draft