AWS Learning Journey

AWS Glue Without the Guesswork: Catalog, Crawlers, and Spark ETL

The Data Catalog, crawlers, databases and tables, PySpark jobs, DynamicFrames, job bookmarks, worker types, triggers, workflows, Data Quality, the Schema Registry, Glue Studio — and how to keep Glue fast and cheap.

8 min read · 1466 words
AWS GlueData EngineeringApache SparkETLAWS
AWS Glue Without the Guesswork: Catalog, Crawlers, and Spark ETL cover image

AWS from Zero to Production — Phase 9, Part 3 of 11 · See the full series roadmap


AWS Glue is one of those services people use for years without quite knowing what it is.

Is it a catalog? A crawler? A Spark service? A visual ETL tool? A data quality engine? A schema registry?

Yes. Glue is a family of capabilities that share one idea: make data in your lake discoverable and transformable without managing servers. The catalog part is used by almost every analytics service on AWS. The ETL part runs serverless Apache Spark.

This article untangles the pieces, then shows you how to write, run, and tune a real Glue job.

What you’ll walk away with


The Glue Data Catalog

The Data Catalog is a central, Hive-compatible metadata store for your data:

The catalog stores metadata only. The data stays in S3.

Why it matters: Athena, Redshift Spectrum, EMR, Glue jobs, and Lake Formation all read the same catalog. Define a table once, and every engine can query it.


Crawlers

A crawler scans a data store — usually an S3 prefix — infers the schema, detects partitions, and creates or updates tables in the catalog.

aws glue create-crawler \
  --name orders-raw-crawler \
  --role GlueCrawlerRole \
  --database-name orders_raw \
  --targets '{"S3Targets":[{"Path":"s3://acme-lake-raw/orders_db/public/orders/"}]}' \
  --schema-change-policy UpdateBehavior=UPDATE_IN_DATABASE,DeleteBehavior=LOG

aws glue start-crawler --name orders-raw-crawler

When crawlers help

When to skip them

Watch out: Crawlers sometimes guess wrong — splitting one dataset into many tables, or inferring string where you wanted timestamp. Point each crawler at one table’s prefix, and check the result.


Glue ETL jobs

A Glue job runs your transformation code on serverless Apache Spark — or as a lightweight Python shell job for small tasks.

A typical PySpark job:

import sys
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from pyspark.sql import functions as F

args = getResolvedOptions(sys.argv, ["JOB_NAME", "OUTPUT_PATH"])
glue = GlueContext(SparkContext.getOrCreate())
spark = glue.spark_session
job = Job(glue)
job.init(args["JOB_NAME"], args)

orders = glue.create_dynamic_frame.from_catalog(
    database="orders_raw",
    table_name="orders",
    transformation_ctx="orders",
).toDF()

curated = (
    orders
    .withColumn("order_date", F.to_date("created_at"))
    .filter(F.col("item").isNotNull())
    .dropDuplicates(["id"])
)

curated.write.mode("append").partitionBy("order_date").parquet(args["OUTPUT_PATH"])

job.commit()

Glue runs Spark 3.x. Pick the latest Glue version for new jobs — it brings newer Spark, Python, and performance improvements.


DynamicFrames vs. DataFrames

dyf = dyf.resolveChoice(specs=[("price", "cast:double")])
df = dyf.toDF()

Use DynamicFrames at the edges — reading messy raw data and writing to catalog-managed targets. Convert to DataFrames for the transformation logic, where Spark’s full API and optimizer shine.


Job bookmarks: process only what’s new

Without bookmarks, a nightly job re-reads all the data every night.

Job bookmarks track what a job has already processed — for S3 sources, which files — so each run picks up only new data.

Watch out: Bookmarks track files, not business keys. If a source rewrites existing files in place, or delivers updates to old records, you need a different strategy — like Iceberg MERGE.


Worker types

Glue capacity is measured in DPUs (data processing units). You choose a worker type and number of workers:

Other options:

Glue bills per DPU-hour, per second, with a short minimum per run. Check current pricing.


Triggers and workflows

Triggers

Start jobs and crawlers:

Workflows

A Glue workflow chains crawlers, jobs, and triggers into a visual DAG:

start (schedule) -> crawler: raw -> job: raw_to_curated -> job: curated_to_gold

Good for pipelines that live entirely inside Glue. For pipelines that span services — DMS, Lambda, Athena, Redshift, notifications — Step Functions or MWAA are usually better (Part 8).


Glue Data Quality

Glue Data Quality checks data with rules written in DQDL (Data Quality Definition Language):

Rules = [
    RowCount > 0,
    IsComplete "id",
    IsUnique "id",
    IsComplete "item",
    ColumnLength "item" <= 200,
    ColumnValues "created_at" <= now()
]

Glue Schema Registry

For streaming data, the Glue Schema Registry stores and versions schemas — Avro, JSON Schema, and Protobuf — and enforces compatibility rules:

It’s how you stop one team’s schema change from silently breaking everyone else’s stream processing.


Glue Studio

Glue Studio is a visual editor for building jobs:

Great for learning and simple pipelines. For production, keep job scripts in Git, and deploy them with IaC and CI/CD.


Cost and performance optimization

orders = glue.create_dynamic_frame.from_catalog(
    database="sales_curated",
    table_name="orders",
    push_down_predicate="order_date >= '2026-10-01'",
)

Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. Where does the Data Catalog store the actual data?
  2. What do job bookmarks track for S3 sources?
  3. Which execution option lowers the cost of a non-urgent nightly job?
  4. How do you read only October’s partitions from a catalog table?

Answers

  1. It doesn’t — it stores metadata. The data stays in S3.
  2. Which files have already been processed.
  3. Flex execution.
  4. A push_down_predicate on the partition column.

What’s next

In Part 4, we query the lake with plain SQL: Amazon Athena — external tables, partitions, partition projection, CTAS, views, workgroups, federation, Iceberg, and keeping costs under control.

Crawlers: love them or avoid them? Tell me in the comments.


Share this article

LinkedIn post draft