AWS Learning Journey

Getting Data In: DMS, Change Data Capture, and Every Way to Ingest on AWS

AWS DMS full loads and CDC, schema conversion, DataSync, Transfer Family, AppFlow, RDS-to-S3 exports, zero-ETL, SaaS ingestion, hybrid migrations, and validation — choosing the right on-ramp for every source.

8 min read · 1583 words
Data EngineeringAWSData MigrationETLChange Data Capture
Getting Data In: DMS, Change Data Capture, and Every Way to Ingest on AWS cover image

AWS from Zero to Production — Phase 9, Part 9 of 11 · See the full series roadmap


A data platform is only as good as the data that reaches it.

And data lives everywhere: a PostgreSQL database behind your app, an Oracle system in the data center, files on a network share, a partner who only speaks SFTP, Salesforce, Zendesk, and a dozen other SaaS tools.

Each source needs a different on-ramp. Pick the wrong one, and you’ll spend months fighting it: nightly dumps that take nine hours, CSVs emailed by partners, or a production database brought to its knees by an analytics query.

This article maps every common source to the right AWS ingestion service — starting with the most important one for databases: AWS Database Migration Service.

What you’ll walk away with


AWS Database Migration Service (DMS)

AWS DMS moves data from databases to databases, data lakes, and warehouses — while the source keeps running.

The pieces

Source DB --(network)--> DMS replication --(write)--> Target (S3 / Redshift / RDS / ...)

Migration types


Full load

DMS reads each selected table and writes it to the target.

Table mappings

Choose what to move — and rename or filter along the way:

{
  "rules": [
    {
      "rule-type": "selection",
      "rule-id": "1",
      "rule-name": "orders-only",
      "object-locator": { "schema-name": "public", "table-name": "orders" },
      "rule-action": "include"
    }
  ]
}

Change data capture (CDC)

CDC reads the source database’s transaction log — PostgreSQL’s WAL, MySQL’s binlog, Oracle’s redo logs — and replicates every insert, update, and delete as it happens.

Why it matters:

Source prerequisites

CDC needs the source configured for it. For example:

CDC to S3

When the target is S3, each change becomes a row in a file with an operation column:

Op, id, item,          created_at
I,  41, coffee beans,  2026-10-31 09:14:02
U,  41, decaf beans,   2026-10-31 09:20:15
D,  12, ,              2026-10-31 09:22:40

Downstream — in Glue, EMR, or Athena with Iceberg — you MERGE these changes into a curated table. That turns a stream of changes into a current-state table you can query.


Schema conversion

Moving between different engines — like Oracle or SQL Server to PostgreSQL or Aurora — means converting schemas, stored procedures, functions, and data types.

For homogeneous migrations — PostgreSQL to PostgreSQL — native tools like pg_dump and logical replication are often simpler.


AWS DataSync: files at scale

AWS DataSync moves files between storage systems — fast and verified:

Use it for: migrating file shares, moving Hadoop data to S3, and recurring transfers from on-premises storage into the lake.


AWS Transfer Family: SFTP without servers

Partners and legacy systems often deliver data via SFTP, FTPS, FTP, or AS2. AWS Transfer Family gives them a managed endpoint that writes directly to S3 or EFS:

Partner --SFTP--> Transfer Family --> s3://acme-lake-raw/partners/acme_logistics/
                                         -> EventBridge -> Step Functions -> Glue

Amazon AppFlow: SaaS data

Amazon AppFlow pulls data from SaaS applications — Salesforce, ServiceNow, Zendesk, Google Analytics, Slack, SAP, and many more — into S3, Redshift, and other targets:

For SaaS tools AppFlow doesn’t support, use the vendor’s export API with Lambda or Glue, or a third-party ingestion tool.


RDS-to-S3 integration

Sometimes you just need a periodic copy of a database in the lake:


SaaS and API ingestion patterns

For sources without a managed connector:

Rules for any custom ingestion:


Hybrid data migration

Moving large datasets from on premises to AWS:


Migration validation

“It copied without errors” isn’t the same as “the data is correct.”


Choosing the right tool

Source                                    Tool
----------------------------------------  ----------------------------------------
Relational DB -> lake, ongoing            AWS DMS (full load + CDC)
Relational DB -> Redshift, ongoing        Zero-ETL integration, or DMS
Relational DB -> another engine           DMS + Schema Conversion
DB snapshot -> lake, periodic             RDS snapshot export to S3
File shares / HDFS -> S3                  AWS DataSync (Snowball if very large)
Partner SFTP / AS2 feeds                  AWS Transfer Family
SaaS apps (Salesforce, Zendesk, ...)      Amazon AppFlow
App events, logs, clickstream             Kinesis / Firehose / MSK (Part 7)
Custom APIs                               Lambda, Glue, or ECS + Scheduler

Common mistakes


Key takeaways


Quick self-check

Test yourself before moving on:

  1. What does CDC read on the source database?
  2. What’s the most common DMS migration type for a live system?
  3. A partner can only send files over SFTP. Which service do you use?
  4. What extra step do CDC files in S3 need before analysts can query current state?

Answers

  1. The transaction log — WAL, binlog, or redo logs.
  2. Full load + CDC.
  3. AWS Transfer Family.
  4. Merge them — applying inserts, updates, and deletes — into a curated table, for example with Iceberg MERGE.

What’s next

In Part 10, we govern all this data: Lake Formation permissions down to columns and rows, data sharing, KMS, Amazon Macie for PII detection, masking, retention, audit logging, lineage, and quality controls.

What’s the hardest data source you’ve ever had to ingest? Tell me in the comments.


Share this article

LinkedIn post draft