Dead-Letter Queue Patterns for CDC Events

A dead-letter queue for CDC events is the difference between a single poison record parking itself for later inspection and that same record silently halting an entire partition of change data. This page, part of event routing and Kafka integration, covers dead-letter design for a change-data pipeline: the Kafka Connect error-tolerance settings that redirect failures, the header enrichment that makes a dead-lettered record diagnosable, how to separate transient from permanent failures, and the redrive tooling and growth alerting that keep a schema break from quietly dropping rows.

Without a dead-letter path, a single unreadable record forces a false choice. Leave errors.tolerance at its default none and the connector task dies on the first bad record, freezing the replication slot it feeds so restart_lsn stops advancing and pg_wal grows until the primary refuses writes. Set errors.tolerance=all without a dead-letter topic and the connector does the opposite — it discards every failing record and logs a line nobody reads, so a serialization break after an ALTER TABLE deletes production changes with no trace. The correct posture is neither halt nor drop: capture the failure with enough context to replay it, alert on the capture rate, and distinguish a failure worth retrying from one that will fail identically forever.

Failure Classification Semantics #

The core decision a CDC dead-letter design encodes is transient versus permanent, because retrying a permanent failure wastes the retry budget and dead-lettering a transient one manufactures false incidents. Transient failures resolve on their own given time; permanent ones fail identically on every replay until code, schema, or data changes. The raw change envelope that these records carry is produced by Python CDC parser development and normalized by JSON to Avro transformation; most permanent failures are a mismatch between that normalized contract and what a consumer expects.

The dangerous middle is a permanent failure misclassified as transient: a deserialization error retried forever pins its partition, and consumer-group lag climbs on that partition alone while its siblings drain — the hot-partition signature covered in alerting on Kafka consumer lag for CDC. Bound retries with a finite count so exhaustion converts a mis-labeled transient into a dead-letter rather than an infinite loop.

Diagnostic Patterns #

The dead-letter topic is only useful if its size and rate are alerted, because a schema break dead-letters records silently — the pipeline stays RUNNING while rows are diverted. Alert on the topic’s production rate, not just its total, so a fresh break pages within minutes.

# Prometheus: page when records are entering the DLQ at all, and hard-page on a burst.
groups:
  - name: cdc-dlq
    rules:
      - alert: CDCDeadLetterGrowing
        expr: sum(rate(kafka_topic_partition_current_offset{topic="orders.events.v1.dlq"}[5m])) > 0
        for: 5m
        labels: {severity: page}
        annotations:
          summary: "Records entering the dead-letter topic — likely a schema or contract break"
      - alert: CDCDeadLetterBurst
        expr: sum(rate(kafka_topic_partition_current_offset{topic="orders.events.v1.dlq"}[5m])) > 100
        for: 2m
        labels: {severity: critical}

Read the enriched headers to triage without reprocessing. The exception.class groups failures so you know whether one schema break or many distinct defects are in play:

# Group DLQ records by failure class from the enriched headers.
kcat -b "$BROKER" -t orders.events.v1.dlq -C -e -f '%h\n' \
  | grep -o '__connect.errors.exception.class=[^,]*' \
  | sort | uniq -c | sort -rn

Correlate the dead-letter onset against the source to prove a schema break rather than bad data. A dead-letter burst that starts exactly when the slot emitted a RELATION change points at an unregistered ALTER TABLE:

-- Publisher: confirm the slot is healthy so a DLQ burst is a contract break, not a stall.
SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots
WHERE slot_type = 'logical';

Safe Deployment Sequence #

Introduce the dead-letter path before you need it, and prove redrive works before an incident makes it load-bearing.

1. Configure tolerance and the dead-letter topic together. Never set errors.tolerance=all without a dead-letter destination, or failures vanish. Enable header context so every captured record is diagnosable:

{
  "errors.tolerance": "all",
  "errors.deadletterqueue.topic.name": "orders.events.v1.dlq",
  "errors.deadletterqueue.topic.replication.factor": "3",
  "errors.deadletterqueue.context.headers.enable": "true",
  "errors.retry.timeout": "60000",
  "errors.retry.delay.max.ms": "5000",
  "errors.log.enable": "true",
  "errors.log.include.messages": "true"
}

2. Pre-create the dead-letter topic. With broker auto-creation disabled, create the topic explicitly with the same replication factor as the main topic so a broker loss does not also lose the failures.

3. Fault-inject a poison record. Produce a deliberately malformed record and confirm it lands in the dead-letter topic with populated __connect.errors.* headers, that the connector stays RUNNING, and that valid records keep flowing. A dead-letter path never exercised is a dead-letter path that fails when it matters.

4. Rehearse a redrive. Fix the injected fault, run the redrive tool against the captured record, and confirm it reprocesses cleanly and idempotently onto the sink. Record the replay throughput so a real backlog has a known drain time.

5. Wire the growth alert to page. Only after redrive is proven, promote CDCDeadLetterGrowing to page. A dead-letter topic without a growth alert is a silent drop with extra steps.

Pipeline Integration #

Redrive is the operation that turns a dead-letter topic from a graveyard into a recovery buffer, and it must be idempotent because a replayed record may have partially applied before it failed. The redrive tool reads the dead-letter topic, uses the enriched headers to reconstruct provenance, and re-produces to the original topic only after the defect is fixed — never blindly, or it re-triggers the same failure.

from confluent_kafka import Consumer, Producer

def redrive(dlq_topic: str, brokers: str, dry_run: bool = True):
    consumer = Consumer({"bootstrap.servers": brokers, "group.id": "dlq-redrive",
                         "enable.auto.commit": False, "auto.offset.reset": "earliest"})
    producer = Producer({"bootstrap.servers": brokers, "enable.idempotence": True})
    consumer.subscribe([dlq_topic])
    while True:
        msg = consumer.poll(2.0)
        if msg is None:
            break
        headers = dict(msg.headers() or [])
        origin = headers.get("__connect.errors.topic", b"").decode()
        exc = headers.get("__connect.errors.exception.class", b"").decode()
        # Replay only classes the fix actually addresses; leave the rest parked.
        if exc not in FIXED_CLASSES:
            continue
        if not dry_run:
            producer.produce(origin, key=msg.key(), value=msg.value())  # LWW sink absorbs replays
        consumer.commit(msg)                                            # advance only on handled records
    producer.flush()

The sink’s last-write-wins upsert from event routing and Kafka integration is what makes replay safe: a redriven record older than current state is absorbed rather than applied, so re-running the dead-letter backlog can never regress the sink. Keep the retry-with-jitter loop bounded — an unbounded retry on a permanent failure never reaches the dead-letter topic, so the growth alert never fires and the break stays invisible. On failover, records already dead-lettered are safe on the broker, but the slot resets; re-establish subscription sync procedures before redriving, so replayed records are not racing a resnapshot of the same rows.

Authoritative references #

• Kafka Connect: error handling and dead-letter queues — errors.tolerance, errors.deadletterqueue.*, and the retry-timeout semantics.
• Confluent: Kafka Connect deep-dive on error handling and DLQs — the enriched __connect.errors.* header set and redrive practice.
• Debezium: handling failures — transform-stage failures and tombstone handling that reach the dead-letter path.

Related #

• Event Routing & Kafka Integration — the parent guide whose idempotent last-write-wins sink makes dead-letter redrive safe to replay.
• Alerting on Kafka Consumer Lag for CDC — the hot-partition lag signature a misclassified permanent failure produces before it is dead-lettered.
• JSON to Avro Transformation — the schema-compatibility contract whose breaks are the most common source of permanent dead-letter failures.