AWS Learning Journey

Data That Never Stops: Streaming on AWS With Kinesis and MSK

Streaming architecture, Kinesis Data Streams, Amazon Data Firehose, Managed Service for Apache Flink, Amazon MSK, producers and consumers, shards and partitions, checkpointing, delivery guarantees, windows, late events, retention, Lambda processing, and schemas.

9 min read · 1730 words
StreamingApache KafkaAWSData EngineeringReal Time Analytics
Data That Never Stops: Streaming on AWS With Kinesis and MSK cover image

AWS from Zero to Production — Phase 9, Part 7 of 11 · See the full series roadmap


A batch pipeline tells you what happened yesterday.

That’s fine for a monthly finance report. It’s useless for fraud detection, a live operations dashboard, inventory that must stay accurate as orders pour in, or an alert that a payment provider just started failing.

Streaming processes data as it happens — in seconds, not hours. It’s powerful, and it’s genuinely harder than batch: data arrives out of order, consumers fall behind, and “process each event exactly once” turns out to be a deep question.

This article covers the AWS streaming toolkit and the concepts you need to use it well.

What you’ll walk away with


Streaming architecture

Producers               Stream                    Consumers
(apps, devices,  --->   (ordered, durable,  --->  (processing, storage,
 CDC, logs)              replayable log)           alerts, ML)

The stream is a durable, ordered, replayable log. Unlike a queue (Phase 4), reading an event doesn’t remove it. Many consumers can read the same stream independently, each tracking its own position — and replay from an earlier point if needed.

Common patterns


Amazon Kinesis Data Streams

Kinesis Data Streams is a fully managed streaming service.

Shards and partition keys

Each shard supports roughly 1 MB/s or 1,000 records/s in, and 2 MB/s out shared across standard consumers.

Watch out: A low-cardinality partition key — like country — creates hot shards, just like hot partitions in DynamoDB.

Capacity modes

Retention

Consumers

aws kinesis put-record \
  --stream-name orders-events \
  --partition-key "order-A17" \
  --data "$(echo -n '{"order_id":"A17","status":"PAID"}' | base64)"

Amazon Data Firehose

Amazon Data Firehose (formerly Kinesis Data Firehose) is the easiest way to deliver streaming data to storage:

Kinesis Data Streams (or direct PUT)
    -> Firehose (buffer, convert to Parquet, partition by date)
    -> s3://acme-lake-raw/events/order_date=2026-10-31/

Firehose delivers in near real time — seconds to minutes, depending on buffering — which is perfect for feeding the lake.


Amazon Managed Service for Apache Flink

For real-time computation — aggregations, joins between streams, pattern detection — you need a stream processor.

Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) runs Apache Flink applications without managing servers:


Amazon MSK: managed Apache Kafka

Amazon Managed Streaming for Apache Kafka (MSK) runs Apache Kafka — the most widely used open-source streaming platform.

Kinesis or MSK?


Producers and consumers

Producers

Consumers


Checkpointing

A checkpoint records how far a consumer has read — the sequence number in Kinesis, or the offset in Kafka.

That last point leads directly to delivery guarantees.


Delivery guarantees

The practical answer: idempotency

Design consumers so processing a record twice has the same effect as once:

Rule: Assume at-least-once, and make consumers idempotent. It’s simpler and more robust than chasing exactly-once everywhere.


Windowing

Streaming aggregations happen over windows of time:

Tumbling (1 min):  |--A--|--B--|--C--|
Sliding (5m/1m):   |-----A-----|
                      |-----B-----|
Session (gap 10m): |-- activity --|   gap   |-- activity --|

Late-arriving events

Events don’t always arrive in order. A phone goes offline, then sends an hour of events at once.

Stream processors like Flink use watermarks — “I believe I’ve seen all events up to time T” — to decide when a window can close, and allowed lateness to accept stragglers.

What to do with very late events:


Stream retention

Retention determines how far back you can replay:


Lambda stream processing

Lambda can consume Kinesis, DynamoDB Streams, MSK, and self-managed Kafka through event source mappings (Phase 4):

Watch out: By default, a failing record blocks its shard — Lambda keeps retrying the same batch until the records expire. Always configure retries, record age limits, and an on-failure destination.


Schema management

Streams outlive the code that writes to them. When a producer changes the shape of events, every consumer is affected.


Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. How do you keep all events for one order in sequence in Kinesis?
  2. Which service delivers a stream into S3 as partitioned Parquet with no code?
  3. A consumer crashes after processing records but before checkpointing. What happens on restart?
  4. What should a Lambda Kinesis consumer have, so one bad record doesn’t block a shard?

Answers

  1. Use the order ID as the partition key, so they land on the same shard.
  2. Amazon Data Firehose.
  3. Those records are processed again — so consumers must be idempotent.
  4. Bisect on error, retry and record age limits, and an on-failure destination.

What’s next

In Part 8, we answer “who runs the pipeline?”: orchestration with Step Functions, MWAA (Apache Airflow), EventBridge Scheduler, and Glue workflows — DAGs, retries, idempotency, backfills, and observability.

Kinesis or Kafka — which does your team run, and why? Tell me in the comments.


Share this article

LinkedIn post draft