AWS Learning Journey

Spark at Scale: Amazon EMR and the Art of Big Data Processing

Apache Spark and the Hadoop ecosystem, EMR on EC2, EMR Serverless, and EMR on EKS, partitioning and shuffles, data skew, memory management, Spot Instances, steps, and troubleshooting.

8 min read · 1600 words
Apache SparkAmazon EMRBig DataData EngineeringAWS
Spark at Scale: Amazon EMR and the Art of Big Data Processing cover image

AWS from Zero to Production — Phase 9, Part 6 of 11 · See the full series roadmap


Your Glue job processed 50 GB in eight minutes. Then the business grew. Now it’s 5 TB, the job runs for four hours, and one task out of two thousand takes three of those hours on its own.

More workers don’t help. The problem isn’t capacity — it’s how the work is divided.

Big data processing is less about big machines and more about understanding how Apache Spark splits, moves, and combines data. This article covers Amazon EMR — AWS’s managed platform for Spark and the Hadoop ecosystem — and the Spark tuning skills that apply wherever Spark runs, including Glue.

What you’ll walk away with


Hadoop ecosystem fundamentals

Apache Hadoop introduced the ideas modern big data still uses:

The wider ecosystem includes Hive (SQL on Hadoop and the metastore that inspired the Glue Data Catalog), Presto and Trino (interactive SQL — the engine behind Athena), HBase (wide-column storage), Flink (streaming), and Hudi and Iceberg (table formats).

EMR can run all of them. Most new work uses Spark.


How Spark works

A Spark application has:

Your code builds a plan of transformations. Nothing runs until an action — like write or count. Then Spark splits the plan into stages at every shuffle, and each stage into tasks — one per data partition.

read (2,000 files) -> filter -> groupBy(customer_id) -> write
|------------- stage 1 -------------|  shuffle  |------ stage 2 ------|
       2,000 tasks                                 200 tasks (default)

Two numbers drive performance:


EMR deployment options

EMR on EC2

A cluster of EC2 instances running Spark, Hive, Trino, and other applications.

Best for: full control, custom configurations, many applications, and steady heavy use.

EMR Serverless

Best for: most new batch Spark jobs — no clusters to manage.

EMR on EKS

Best for: organizations standardized on Kubernetes that want to share infrastructure.

Where Glue fits

AWS Glue also runs serverless Spark, with the catalog, bookmarks, and Data Quality built in. Rough guide:


Partitioning and shuffling

Input partitions

Spark creates partitions from input files — roughly one per file, or per chunk of a large splittable file. Thousands of tiny files mean thousands of tiny tasks (the small-file problem from Part 2).

Shuffle partitions

After a shuffle — from groupBy, join, distinct, or repartition — the number of partitions is set by spark.sql.shuffle.partitions (default 200).

Adaptive Query Execution (AQE) — on by default in modern Spark — fixes much of this at runtime: it coalesces small shuffle partitions, switches join strategies, and splits skewed partitions.

Avoid shuffles where you can

from pyspark.sql.functions import broadcast

enriched = orders.join(broadcast(products), "product_id")

Data skew

Skew is when a few partitions hold far more data than the rest. One task processes 50 GB while the others process 50 MB — and the whole stage waits for the slow one.

How to spot it

In the Spark UI, a stage where most tasks finish in seconds but a few take minutes — with much larger “shuffle read size.”

Common causes

Fixes

from pyspark.sql import functions as F

salted = orders.withColumn("salt", (F.rand() * 16).cast("int"))
partial = salted.groupBy("customer_id", "salt").agg(F.sum("total").alias("t"))
result = partial.groupBy("customer_id").agg(F.sum("t").alias("total"))

Memory management

Each executor’s memory is split between execution (shuffles, joins, sorts, aggregations) and storage (cached data), plus overhead for the JVM and Python processes.

Symptoms of memory trouble:

Fixes, roughly in order:

  1. Reduce data early — filter rows, select fewer columns.
  2. Increase partitions so each task handles less data.
  3. Avoid collect() and toPandas() on large data — they pull everything onto the driver.
  4. Avoid Python UDFs where built-in functions exist — they’re slower and use extra memory. Use pandas UDFs if you must.
  5. Give executors more memory — or overhead, for PySpark — or choose memory-optimized instances.

Spot Instances for EMR

Spot (Phase 4) can cut EMR compute costs substantially:


Step orchestration

On EMR on EC2, a step is a unit of work — a Spark application, Hive script, or custom JAR:

aws emr add-steps \
  --cluster-id j-ABC123 \
  --steps Type=Spark,Name=curate-orders,ActionOnFailure=CONTINUE,\
Args=[--deploy-mode,cluster,s3://acme-code/jobs/curate_orders.py,--date,2026-10-31]

Logging and troubleshooting

A quick troubleshooting guide


Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. What triggers a new stage in Spark?
  2. One task in a stage takes 40 minutes while others take 30 seconds. What’s likely wrong?
  3. Which EMR node type is safest to run on Spot?
  4. When does a broadcast join help?

Answers

  1. A shuffle — from operations like groupBy, join, or repartition.
  2. Data skew — one partition has far more data than the others.
  3. Task nodes.
  4. When one side of the join is small enough to send to every executor, avoiding a shuffle of the large side.

What’s next

In Part 7, data stops waiting for a schedule: streaming with Kinesis Data Streams, Amazon Data Firehose, Managed Service for Apache Flink, and Amazon MSK — shards, checkpoints, delivery guarantees, windows, and late events.

What’s the worst Spark skew you’ve debugged? Share it in the comments.


Share this article

LinkedIn post draft