AWS Learning Journey

Data Lake or Data Swamp? Designing an S3 Data Lake That Stays Clean

Zones, prefix design, date partitioning, Parquet, ORC, and Avro, compression, the small-file problem, lifecycle rules, encryption, cross-account access, and event-driven pipelines.

8 min read · 1461 words
Data EngineeringAWSData LakeAmazon S3Big Data
Data Lake or Data Swamp? Designing an S3 Data Lake That Stays Clean cover image

AWS from Zero to Production — Phase 9, Part 2 of 11 · See the full series roadmap


Every data lake starts clean.

Six months later, there are folders called final, final_v2, and final_v2_USE_THIS. Nobody knows which files are raw and which are cleaned. A query that should take seconds scans 40 TB. A million tiny JSON files make every job crawl. And the bill grows every month because nothing is ever deleted.

That’s a data swamp, and it’s the default outcome without design.

The fix isn’t a fancy tool. It’s a handful of conventions — zones, prefixes, partitions, formats, and lifecycle rules — decided up front and enforced everywhere. This article gives you those conventions.

What you’ll walk away with


Data lake zones

A zone is a stage of data refinement. Most lakes use three:

Raw (bronze, landing)

Curated (silver, cleaned)

Consumption (gold, analytics)

Source -> RAW (as-is) -> CURATED (clean, typed, Parquet) -> CONSUMPTION (business-ready)

Engineer’s note: Many teams use separate buckets per zone — like acme-lake-raw and acme-lake-curated — so bucket policies, encryption keys, lifecycle rules, and replication can differ by zone.


S3 prefix design

Decide a naming pattern once, and enforce it:

s3://acme-lake-raw/{source_system}/{dataset}/ingest_date=YYYY-MM-DD/{files}
s3://acme-lake-curated/{domain}/{table}/{partition_column}=value/{files}
s3://acme-lake-consumption/{use_case}/{table}/{partition_column}=value/{files}

Examples:

s3://acme-lake-raw/orders_db/public.orders/ingest_date=2026-10-31/LOAD00000001.parquet
s3://acme-lake-curated/sales/orders/order_date=2026-10-31/part-00000.snappy.parquet
s3://acme-lake-consumption/finance/daily_revenue/month=2026-10/part-00000.snappy.parquet

Rules worth following:


Date-based partitioning

Most analytics filters by time, so most tables partition by date.

Which date?

How granular?

order_date=2026-10-31                  daily: the common default
year=2026/month=10/day=31              nested: works, but harder to filter on ranges
event_hour=2026-10-31-14               hourly: for very high volume

Tip: A single date column like order_date=2026-10-31 is easier to query (WHERE order_date BETWEEN ...) than separate year, month, and day columns.


File formats

Parquet

ORC

Avro

Open table formats

Apache Iceberg — supported by Athena, Glue, EMR, and Redshift — adds a metadata layer on top of Parquet files:

For curated tables that change — late updates, deletes for privacy requests, CDC — Iceberg is increasingly the default.


Compression

Rule: Compress everything. Storage, scan cost, and network transfer all drop.


The small-file problem

Thousands of tiny files are far slower than a few big ones — even with the same total data.

Why:

Where small files come from

How to fix it


S3 lifecycle management

From Phase 5, applied to a lake:

Raw zone:
  Standard -> Standard-IA after 30 days -> Glacier Instant Retrieval after 90 days
  Expire after 2 years (or keep, if reprocessing and compliance require it)

Curated zone:
  Intelligent-Tiering (access patterns vary by table)

Query results and temporary data:
  Expire after 7 days

Everywhere:
  Abort incomplete multipart uploads after 7 days
  Expire noncurrent versions after 30 days

Tag objects or use prefixes per zone, so rules apply cleanly.


Data lake encryption


Cross-account access

Large organizations run the lake in a central data account and let consumer accounts — analytics, ML, and business units — read from it.

Options:

Remember: with SSE-KMS, the KMS key policy must also allow the consumer account.


Event-driven pipelines

S3 can trigger processing as data arrives, instead of waiting for a schedule:

New object in s3://acme-lake-raw/partner_feed/
   -> S3 event notification to EventBridge
   -> EventBridge rule
   -> Step Functions: validate -> Glue job -> update catalog -> notify

Watch out: Event-driven pipelines run once per file. If a source drops 10,000 files, you get 10,000 events. Batch them — with SQS and a scheduled or size-based trigger — when per-file processing would be wasteful.


Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. Why should the raw zone be immutable?
  2. What’s wrong with partitioning a table by hour when each hour has 50 KB of data?
  3. Which format suits streaming landing data: Avro or Parquet?
  4. A consumer account can list objects but gets AccessDenied reading them, and the bucket uses SSE-KMS. What’s missing?

Answers

  1. So you can reprocess from the original data when logic changes or bugs are found.
  2. It creates the small-file problem — huge overhead for very little data.
  3. Avro — row-based, compact, with an embedded schema.
  4. Permission in the KMS key policy — and the consumer’s IAM policy — for the key.

What’s next

In Part 3, we meet the workhorse of AWS data engineering: AWS Glue — the Data Catalog, crawlers, PySpark ETL jobs, DynamicFrames, bookmarks, worker types, triggers, workflows, and Data Quality.

What’s the strangest folder name you’ve found in a data lake? Share it in the comments.


Share this article

LinkedIn post draft