Alerting on Kafka Consumer Lag for CDC

Kafka consumer-lag alerting for a CDC pipeline has to answer one question a raw lag number cannot: whether a growing backlog lives in PostgreSQL, in the Kafka Connect worker, or in the sink consumer — because the remediation for each is completely different. This page, an extension of event routing and Kafka integration, defines the broker-side lag signals for a change-data pipeline, the Debezium and Connect task metrics that sit beside them, and the correlation against the replication slot’s confirmed_flush_lsn that localizes a stall to a single stage.

Alert on the wrong signal and you page the wrong team at 3 a.m. A rising consumer-group lag looks identical whether the database is retaining WAL, the connector task has died, or a sink consumer is wedged — but restarting the connector when the real problem is a slow sink does nothing, and scaling consumers when the connector is down does less. Worse, lag alerting built on a single instantaneous threshold flaps constantly: every deploy, rebalance, or nightly batch spikes lag briefly, so an alert that fires on any breach trains on-call to ignore it. The design here separates the signals by stage and gates every alert on lag that is both above a threshold and sustained, so a page means a real, localized stall.

Lag Signal Semantics #

Consumer-group lag on a CDC topic is not one number but a family of signals emitted at different stages, each answering a different question. Treat them as a chain: the slot metrics cover everything up to the walsender, and once changes cross into Kafka the observability continues here — the split is the same one drawn in async monitoring integration. The table maps each signal to what it measures and what a breach localizes.

The trap that this table exists to close is the dead-task false-green. A FAILED connector task produces zero new records, so its topic’s log-end offset stops moving, so consumer-group lag stops growing and even shrinks as the sink drains the backlog — the lag dashboard goes green while replication is fully stopped. Alert on task state and on the slot’s restart_lsn freezing, not on group lag alone, or a stalled pipeline will read as healthy.

Diagnostic Patterns #

Every alert below is gated on rising and sustained, expressed in PromQL as a positive slope over a window combined with an absolute floor. A bare > threshold fires on every deploy spike; the compound form fires only on a genuine, growing backlog.

# Prometheus alert: per-partition consumer lag that is both high and still climbing.
groups:
  - name: cdc-kafka-lag
    rules:
      - alert: CDCConsumerLagGrowing
        expr: |
          kafka_consumergroup_lag{group="orders-sink-v1"} > 100000
          and
          deriv(kafka_consumergroup_lag{group="orders-sink-v1"}[10m]) > 0
        for: 10m
        labels: {severity: page, stage: sink-consumer}
        annotations:
          summary: "Sink consumer lag high and rising on {{ $labels.topic }}/{{ $labels.partition }}"

      # Connector trailing the source commit — localizes to Kafka Connect, not the sink.
      - alert: DebeziumBehindSource
        expr: |
          debezium_metrics_MilliSecondsBehindSource{context="streaming"} > 60000
          and
          deriv(debezium_metrics_MilliSecondsBehindSource{context="streaming"}[10m]) > 0
        for: 10m
        labels: {severity: page, stage: connector}
      # A task that is not RUNNING produces zero records — page regardless of lag.
      - alert: ConnectTaskNotRunning
        expr: kafka_connect_connector_task_status{status!="running"} == 1
        for: 2m
        labels: {severity: page, stage: connector}

The single query that localizes a stall is the correlation between the broker offset lag and the slot’s confirmed_flush_lsn. If group lag is high but the slot is current, the database is fine and the sink is the problem; if the slot is behind too, the stall is upstream of Kafka entirely:

-- Publisher: is the slot keeping up, or is the whole pipeline stalled upstream of Kafka?
-- If retained_wal is small while Kafka group lag is large -> problem is downstream (sink).
-- If retained_wal is large -> problem is the DB producer or the connector read side.
SELECT slot_name,
       active,
       confirmed_flush_lsn,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS flush_lag,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))         AS retained_wal
FROM pg_replication_slots
WHERE slot_type = 'logical';

# Ground-truth the exporter against Kafka directly when a number looks wrong.
kafka-consumer-groups.sh --bootstrap-server "$BROKER" \
  --describe --group orders-sink-v1
# LAG column per partition; a single hot partition with all others at zero is a
# routing/hot-key problem, not a consumer-capacity problem.

Safe Deployment Sequence #

Roll out lag alerting so the thresholds are calibrated to real traffic before they gate a page, not guessed.

1. Deploy the exporters read-only. Stand up kafka-lag-exporter or Burrow and the Debezium JMX exporter, and confirm kafka_consumergroup_lag, MilliSecondsBehindSource, and task state all appear in Prometheus. Verify the numbers against kafka-consumer-groups.sh so you trust the source.

2. Baseline a week of normal lag. Record peak-hour group lag, the normal MilliSecondsBehindSource band, and typical deploy/rebalance spikes. The absolute floor in each alert should sit above the routine spike, not at zero.

3. Add the compound alerts in warn-only. Ship the rising AND above floor rules at severity: warn first and watch for a week. Tune the for: duration up if rebalances trip it; tune the floor up if nightly batches do.

4. Promote to page and add the dead-task guard. Once the warn tier is quiet on healthy traffic, promote the sink-lag and connector-lag alerts to page, and keep ConnectTaskNotRunning and the slot restart_lsn-frozen alert at page unconditionally — those detect the false-green a lag-only alert misses.

5. Wire the localization runbook. Each page links to the correlation query above so on-call reads slot lag before touching Kafka: high slot lag routes to the database/connector runbook, current slot with high group lag routes to sink scaling.

Pipeline Integration #

Lag alerting is only useful if a page maps to an action, so the on-call automation reads the same signals in the same order the localization rule prescribes. A small correlator that queries the slot with asyncpg and the group lag from Prometheus can annotate the alert with the localized stage before a human looks at it:

import asyncpg, httpx

async def localize(dsn: str, prom: str, group: str) -> str:
    conn = await asyncpg.connect(dsn)
    try:
        flush_lag = await conn.fetchval("""
            SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)
            FROM pg_replication_slots WHERE slot_type = 'logical'
            ORDER BY 1 DESC LIMIT 1""")
    finally:
        await conn.close()

    q = f'sum(kafka_consumergroup_lag{{group="{group}"}})'
    async with httpx.AsyncClient() as c:
        r = await c.get(f"{prom}/api/v1/query", params={"query": q})
    group_lag = float(r.json()["data"]["result"][0]["value"][1])

    if flush_lag > 256 * 1024 * 1024:     # slot > 256 MB behind
        return "upstream: PostgreSQL producer or connector read side"
    if group_lag > 100_000:               # slot current, broker backlog large
        return "downstream: sink consumer capacity"
    return "no localized stall (transient)"

For the sink consumer itself, the durable fix for recurring lag is capacity that matches the ordering domain, not a bigger poll batch: raise partition count on a new topic version so more consumers can run in parallel, using the partition-key design from event routing and Kafka integration. Where lag is caused by a poison record blocking a partition rather than by throughput, the answer is to route it aside rather than let it wedge the group — the mechanics live in dead-letter queue patterns for CDC events. On failover, a rebuilt slot resets confirmed_flush_lsn, so silence the correlation alert until subscription sync procedures re-establish a known-good LSN, or the localizer will read the resnapshot as an upstream stall.

Authoritative references #

• Debezium: monitoring and metrics — MilliSecondsBehindSource, SourceRecordActiveCount, and the streaming/snapshot JMX MBeans.
• Apache Kafka: consumer lag and kafka-consumer-groups — the log-end-offset-minus-committed-offset definition of lag.
• Kafka Connect: monitoring REST and task status — connector/task state and error metrics behind the dead-task alert.
• PostgreSQL: pg_replication_slots — confirmed_flush_lsn and restart_lsn for the correlation query.

Related #

• Event Routing & Kafka Integration — the parent guide on topic naming, partition-key design, and the ordering guarantees whose breach shows up as skewed per-partition lag.
• Dead-Letter Queue Patterns for CDC Events — how a poison record that would otherwise pin a partition’s lag is routed aside instead.
• Async Monitoring Integration — the database-side lag signals this page correlates against, and where the slot metrics originate.