Microsoft Azure Learning Journey

Spark on Azure: Databricks, Delta Lake, and Compute Choices

Understand Spark's distributed execution, choose compute for the job, and govern lake data with Delta Lake and Unity Catalog.

4 min read · 734 words
Microsoft AzureData EngineeringAnalyticsData GovernanceCloud Computing
Spark on Azure: Databricks, Delta Lake, and Compute Choices cover image

Apache Spark can transform data across many files and machines, but adding a cluster does not make a pipeline fast or correct. Workload shape, partitioning, shuffle, file layout, and access controls determine whether a job scales. On Azure, teams can run Spark through Azure Databricks, Synapse Spark, or Microsoft Fabric; the choice changes the workspace, governance, and compute model.

What you’ll learn

Prerequisites

Understand the Spark job

A Spark application has a driver that plans work and coordinates executors that process partitions. Transformations build a logical plan; actions execute it. Operations that move data across partitions—such as wide aggregations, joins, and sorts—cause shuffles, which consume network, disk, and memory.

Use the execution plan and job metrics to find skew, spills, long tasks, and oversized shuffles. Avoid collecting a large dataset to the driver with collect(). Avoid creating thousands of tiny partitions or one enormous partition. Choose partition counts and join strategies from the data size and cluster shape, then measure representative runs.

Use Delta Lake intentionally

Delta Lake stores table data as Parquet with a transaction log that coordinates table versions and supported transactional operations. It supports reliable batch and streaming patterns, schema controls, and time travel within the table’s retention and feature constraints. A directory of Parquet files without the Delta transaction log does not acquire those table guarantees merely because Spark can read it.

from pyspark.sql import functions as F

orders = (
    spark.read.format("parquet")
    .load("abfss://raw@<storage-account>.dfs.core.windows.net/orders/")
    .withColumn("order_date", F.to_date("created_at"))
    .filter(F.col("order_id").isNotNull())
)

(orders.write
    .format("delta")
    .mode("append")
    .partitionBy("order_date")
    .save("abfss://curated@<storage-account>.dfs.core.windows.net/orders/"))

This example is illustrative: it does not deduplicate, quarantine bad timestamps, or make append idempotent. Production logic needs a stable key, explicit schema, late-data policy, and a tested overwrite/merge strategy. Partitioning by a high-cardinality field such as order ID can create excessive directories; choose partitions from query filters and file volume.

Choose compute for the work

Interactive compute supports exploration; job-oriented compute isolates scheduled workloads and can terminate after completion. Serverless and managed compute options vary by workspace, region, account, and workload. Confirm supported access modes and policies before building around a particular option. Use autoscaling and auto-termination carefully, pin runtime dependencies, and measure startup time as well as job duration.

Separate development from production data and credentials. Use the organization’s approved identity path, such as a Unity Catalog storage credential backed by an Azure Databricks Access Connector managed identity, and scope the storage role to the required container. Avoid storage account keys and shared secrets in notebooks or cluster configuration. Unity Catalog’s grants govern catalog objects; the storage credential and underlying Azure role govern access to external data. Both sides must be configured.

Choose the engine, not just the language

Compare supported Spark runtimes, libraries, networking, data access, catalog behavior, job orchestration, capacity/compute cost, and operational ownership. A common language does not make the engines or table features identical.

✅ Checkpoint: For a slow Spark job, identify its input size, partition count, shuffle stages, skew, output-file size, storage identity, and measured cost per successful dataset run.

Next: Stream data with Event Hubs.

Series navigation: Previous: Synapse or Fabric · Current: Phase 9, Part 6 · Next: Event Hubs streaming

References

Share this article

LinkedIn post draft