Configuring Debezium Snapshot Modes

Debezium's snapshot.mode decides whether the PostgreSQL connector backfills existing rows before it streams WAL, and the wrong value resnapshots a…

Debezium’s snapshot.mode decides whether the PostgreSQL connector backfills existing rows before it streams WAL, and the wrong value resnapshots a billion-row table on every restart or silently drops the history a consumer needed. This page sits under Debezium connector configuration and isolates that one property: what each mode does to the initial read and the streaming handoff, how the snapshot phase interacts with slot and WAL retention, the lock behavior you accept with each locking mode, and how to resnapshot a single table on demand without stopping the connector. Every mode is validated against PostgreSQL 14–17 and Debezium 2.x with the pgoutput plugin.

The cost of a snapshot decision is paid at the worst possible time — a restart during an incident. A mode that resnapshots re-reads every captured table under a consistent transaction, holding the slot’s restart_lsn frozen so pg_wal grows for the entire scan, while the primary keeps taking writes. On a large table that scan can run for hours and threaten the disk before a single new change event flows. Conversely a mode that never snapshots will start a fresh consumer mid-stream with no baseline, so the downstream store is missing every row that existed before the connector first connected. Neither failure is loud at deploy time; both surface later as a full disk or a silent data gap.

Debezium snapshot modes across the snapshot and streaming phases A three-column grid with a header row and five mode rows. The first column names the mode, the second describes the snapshot phase, the third describes the streaming phase. initial: green backfill all rows once, green then stream WAL. initial_only: green backfill all rows once, red stop with no streaming. never: red no backfill, green stream new WAL only. when_needed: amber backfill only if offset or slot is gone, green then stream WAL. no_data: amber capture schema but no rows, green then stream WAL. A footnote notes that incremental snapshots are triggered on demand through a signal, independent of the startup mode. snapshot.mode Snapshot phase Streaming phase initial backfill all rows once then stream WAL initial_only backfill all rows once stop — no streaming never no backfill stream new WAL only when_needed backfill only if offset/slot gone then stream WAL no_data capture schema, no rows then stream WAL Incremental snapshots are separate: triggered on demand through a signal, independent of the startup mode above.
Every startup mode is one snapshot decision plus one streaming decision; when_needed and no_data are the conditional cases that make failover and re-seed handling predictable.

Snapshot Mode Semantics

The snapshot runs inside a single repeatable_read transaction so the backfilled rows form a consistent point-in-time set, and the streaming phase then picks up every WAL change committed after that point — the handoff the Debezium connector configuration guide diagrams as the snapshot-to-streaming transition. The mode chooses which half of that handoff runs. The table below is the full behavior matrix for the Postgres connector.

snapshot.mode Snapshot behavior Streaming behavior When to use
initial (default) Backfills every captured table once, on first start only Streams WAL after the snapshot completes The standard first deployment — a full baseline plus live changes.
initial_only Backfills every captured table once Stops; never streams One-shot bulk load into a sink, then the connector is retired. Pair with a separate streaming connector if you later need live changes.
never No backfill Streams new WAL from the slot’s current position The sink already holds the baseline (e.g. seeded by a separate load) and you only want changes from now on.
when_needed Backfills only if the stored offset is missing or the slot’s restart_lsn is unreachable Streams after any needed snapshot Failover and disaster recovery: resnapshots automatically only when history is truly lost, otherwise resumes.
no_data (formerly schema_only) Captures table schema but reads no rows Streams new WAL only You want change events with correct schema from now on, but do not need the pre-existing rows.
always Backfills every captured table on every start Streams after each snapshot Rare — test rigs or tables small enough that a fresh baseline each start is cheap and desirable. Dangerous on large tables.

Two behaviors are easy to misjudge. First, the snapshot phase freezes the slot: while the consistent read is open the restart_lsn cannot advance, so WAL accumulates for the whole scan, and on a multi-hour snapshot that retention can approach max_slot_wal_keep_size and threaten the disk. Second, initial does not resnapshot on an ordinary restart — it snapshots only when there is no committed offset, which is why a routine connector bounce with slot.drop.on.stop=false resumes streaming instead of re-reading the table. A full resnapshot happens only when the offset and the slot are gone, which is exactly the condition when_needed detects and handles gracefully.

Lock behavior is governed separately by snapshot.locking.mode. minimal (the default) takes table locks only for the brief initial catalog read and then relies on repeatable_read isolation for the row scan — lowest production impact. extended holds locks across the whole snapshot, guaranteeing consistency against concurrent DDL but potentially blocking writers for the scan duration. none takes no locks and is safe only on append-only or read-only tables, because concurrent DDL during an unlocked snapshot can corrupt the captured schema.

Diagnostic Patterns

The snapshot phase is visible on the PostgreSQL side as a frozen slot and a long-lived repeatable_read transaction. Watch both while a snapshot runs.

sql
-- Publisher: during a snapshot, restart_lsn is frozen and retained_wal grows.
SELECT slot_name,
       active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))         AS retained_wal,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unconfirmed
FROM pg_replication_slots
WHERE slot_name = 'debezium_pg_analytics';
-- retained_wal climbing with restart_lsn static = snapshot in progress (or a stalled snapshot).
sql
-- Publisher: find the connector's long-running snapshot transaction and its age.
SELECT pid, state, backend_xmin,
       now() - xact_start AS txn_age,
       left(query, 60)    AS current_query
FROM pg_stat_activity
WHERE application_name LIKE '%debezium%'
ORDER BY xact_start;
-- A snapshot holds one long repeatable_read txn; a very old backend_xmin also blocks vacuum.

Confirm which phase the connector reports, so you can tell a still-snapshotting connector from a stalled one.

bash
# Debezium exposes snapshot progress and phase over JMX.
kafka-run-class kafka.tools.JmxTool \
  --object-name 'debezium.postgres:type=connector-metrics,context=snapshot,server=pg-analytics,*' \
  --attributes SnapshotRunning,SnapshotCompleted,RowsScanned,RemainingTableCount
  • retained_wal climbing with restart_lsn frozen for hours — a snapshot is running (expected) or wedged; check RemainingTableCount and disk headroom against max_slot_wal_keep_size.
  • backend_xmin far behind xact_start old — the snapshot transaction is holding back vacuum; large snapshots bloat the primary, so schedule them off-peak.
  • SnapshotRunning=false and SnapshotCompleted=false with no streaming — the connector never started its snapshot; check task state and the snapshot.mode value.

Safe Deployment Sequence

Pick the mode for the outcome you need, and use an incremental snapshot to backfill new tables instead of forcing a full resnapshot of everything.

1. First deployment — full baseline. Start at initial and let the snapshot complete before trusting downstream counts. Watch retained_wal against the disk while it runs.

json
{ "snapshot.mode": "initial", "snapshot.locking.mode": "minimal",
  "snapshot.isolation.mode": "repeatable_read", "snapshot.fetch.size": "10240" }

2. Add a table without resnapshotting the rest. Set up the signaling channel once, then send an execute-snapshot signal that incrementally backfills only the new table in chunks, interleaved with live streaming, so nothing else re-reads.

json
{ "signal.data.collection": "public.debezium_signal",
  "incremental.snapshot.chunk.size": "1024" }
sql
-- Trigger an incremental snapshot of one newly added table, on demand.
INSERT INTO public.debezium_signal (id, type, data)
VALUES (gen_random_uuid()::text, 'execute-snapshot',
        '{"data-collections": ["public.new_table"], "type": "incremental"}');

3. Re-seed after history is lost. If the slot was invalidated or the offset dropped, switch to when_needed and re-PUT the config; it resnapshots only because the history is genuinely gone, then resumes streaming.

4. Revert. Snapshot mode is read at connector start, so a change takes effect on the next restart. Reverting is re-PUT-ing the prior snapshot.mode; because the committed offset survives, switching back to initial on an already-snapshotted connector does not re-trigger a snapshot.

Pipeline Integration

Snapshot mode is where extraction meets failover planning, because the mode determines whether a promoted primary forces a full resnapshot or resumes cleanly. Before PostgreSQL 16, logical slots did not fail over — they lived only on the old primary — so a promotion left the connector with no slot and, under initial, only a resnapshot could recover; under when_needed that resnapshot happens automatically. On PG 16/17 a failover-enabled slot synchronized to the standby lets the connector resume streaming after promotion with no snapshot at all. The full recovery procedure, including slot synchronization and how to reconcile the offset against the new primary’s restart_lsn, is covered in failover and slot recovery.

  • Idempotent, LSN-guarded apply. A resnapshot replays rows the sink may already hold, so consumers must upsert on the primary key and ignore any row whose source LSN or ts_ms is not newer than the stored high-water mark — the pattern from Python CDC parser development. This is what makes when_needed safe: a spurious resnapshot converges instead of duplicating.
  • Per-table strategy. Choose the mode per capture surface. Small dimension tables tolerate initial or even always; a billion-row fact table should be baselined once with initial and thereafter grown with incremental snapshots, never re-read wholesale.
  • Retry with backoff. Wrap the signal-and-monitor loop so a transient registry or broker error during an incremental snapshot backs off and resumes rather than reissuing the signal, which would restart the chunked scan from the beginning.
  • Guard the disk during snapshots. Because the snapshot freezes restart_lsn, alert on retained_wal against max_slot_wal_keep_size for the whole scan; a snapshot that outruns retention invalidates its own slot and forces it to start over.

The change events a snapshot emits flow onward through the same serialization and routing stages as live changes — the JSON to Avro transformation contract and the partitioning in event routing and Kafka integration apply identically to snapshot and streaming records.

Authoritative references