Writing Alert Rules for Slot and WAL Thresholds

Prometheus alerting rules for logical replication encode five failure modes as expr/for/severity triples — an inactive slot, retained WAL crossing a fraction of max_slot_wal_keep_size, wal_status leaving reserved, apply lag sustained past a window, and the exporter gone silent — and getting the thresholds and hold-downs right is what makes the Monitoring & Alerting pipeline page on real disk risk instead of flapping on noise. A rule set that omits the exporter-absent case, or that fires on an instantaneous byte spike with no for clause, either misses the outage or trains the on-call team to ignore the pager.

This guide is about the rule YAML specifically: the groups:/rules: structure, choosing expr against the exported gauges, the for duration that suppresses flapping, the labels.severity that routes the page, and annotations that make the alert actionable at 3 a.m. It assumes the gauges are already exported and scraped.

Alert Semantics #

Each rule maps one replication failure to a threshold, a hold-down, and a severity. The table is the specification the YAML implements; every row below corresponds to one rule in the group that follows. Thresholds are expressed against the exported gauges so they move with configuration rather than drifting as hard-coded constants.

Two design choices are load-bearing. The retained-bytes rules divide by the exported cap rather than comparing against a literal byte count, so changing max_slot_wal_keep_size never silently breaks the alert — the threshold is a fraction, not a number. And ExporterDown is critical, not informational, because its absence is what lets every other rule read as healthy while replication is broken; the failure to alert on up == 0 is the most common gap in a replication rule set. The mechanics of when a slot freezes restart_lsn — the condition LogicalSlotInactive detects — are covered in initializing replication slots.

Diagnostic Patterns #

The complete rule group encodes every row above. Group related rules so they share an evaluation interval, and keep the expr readable — a rule no on-call engineer can parse under pressure is a rule that gets silenced.

# /etc/prometheus/rules/replication.yml
groups:
  - name: logical-replication
    rules:
      - alert: LogicalSlotInactive
        expr: pg_repl_slot_lag_active == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Logical slot {{ $labels.slot_name }} has no active consumer"
          description: "restart_lsn is frozen; WAL is pinning on the primary. Restore the consumer or drop the slot."

      - alert: SlotRetainedBytesHigh
        expr: >
          pg_repl_slot_lag_retained_bytes
            / clamp_min(pg_repl_slot_wal_keep_bytes, 1) > 0.8
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Slot {{ $labels.slot_name }} retains >80% of max_slot_wal_keep_size"
          description: "Slot invalidation is imminent. Recover the consumer now to avoid a resnapshot."

      - alert: SlotWalStatusNotReserved
        expr: pg_repl_slot_lag_reserved == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Slot {{ $labels.slot_name }} wal_status left 'reserved'"
          description: "WAL beyond the cap is being dropped; the slot may already be unrecoverable."

      - alert: ApplyLagSustained
        expr: pg_repl_slot_lag_unconfirmed_bytes > 256 * 1024 * 1024
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Slot {{ $labels.slot_name }} apply lag >256 MB for 10m"
          description: "Consumer is falling behind. Check ETL batch sizing and network to the subscriber."

      - alert: ExporterDown
        expr: up{job="postgres"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "postgres_exporter target {{ $labels.instance }} is down"
          description: "Replication metrics are stale. Every slot alert is blind until this clears."

Before trusting a rule in production, evaluate its expr in the Prometheus expression browser and confirm it returns the series you expect at the boundary. A rule whose expression returns no series never fires — the silent-failure mode that mirrors a dead exporter.

# Prove the retained-fraction expression returns a value between 0 and 1 per slot.
pg_repl_slot_lag_retained_bytes / clamp_min(pg_repl_slot_wal_keep_bytes, 1)

Use promtool to unit-test the group so a future edit cannot regress a threshold undetected. A unit test asserts that, given a synthetic series, the rule fires at the right minute with the right labels — which catches an off-by-one in a for duration or a broken matcher before it reaches production.

# replication_tests.yml — assert SlotRetainedBytesHigh fires at 82% of cap after 5m.
rule_files: [replication.yml]
tests:
  - interval: 1m
    input_series:
      - series: 'pg_repl_slot_lag_retained_bytes{slot_name="orders"}'
        values: '0+9e9x10'                       # climbs past 80% of a 10 GB cap
      - series: 'pg_repl_slot_wal_keep_bytes'
        values: '10737418240x10'
    alert_rule_test:
      - eval_time: 8m
        alertname: SlotRetainedBytesHigh
        exp_alerts:
          - exp_labels: {severity: critical, slot_name: orders}

# Validate rule syntax and run alert unit tests in CI before shipping.
promtool check rules /etc/prometheus/rules/replication.yml
promtool test rules replication_tests.yml

Safe Deployment Sequence #

Roll rules out so a bad threshold is caught before it pages, and revert is a single reload.

1. Author and lint locally. Write the group, run promtool check rules, and confirm every expr parses and every referenced metric name exists in your Prometheus.

2. Ship at low severity first. Deploy new rules as severity: warning routed to a chat channel, not the pager, and watch them for a full traffic cycle. A rule that fires overnight on idle-system quirks — time-based lag reading zero, or a slot briefly inactive during a planned restart — needs its for window widened before it earns pager status.

3. Reload Prometheus, no restart. Rule files load on a config reload, so promotion is non-disruptive.

# Reload rules without dropping the scrape history.
curl -X POST http://prometheus.internal:9090/-/reload
# or: kill -HUP $(pidof prometheus)

4. Wire Alertmanager routing by severity. Map the severity label to a receiver so critical alerts page and warnings post. Set repeat_interval short for slot bloat, because a retained-bytes alert that re-pages every four hours is too slow when the disk is filling.

# alertmanager.yml — route replication severities to the right place.
route:
  receiver: chat
  group_by: ['alertname', 'slot_name']
  routes:
    - matchers: [severity="critical"]
      receiver: pager
      repeat_interval: 30m         # slot bloat needs re-paging faster than the 4h default
receivers:
  - name: pager
    # pagerduty_configs / opsgenie_configs ...
  - name: chat
    # slack_configs ...

Revert is removing the rule from the file and reloading; because rules only observe series, disabling one never affects replication. Keep the rule file, the dashboard JSON, and the exporter’s queries.yaml in one repository so a threshold, the panel that shows it, and the gauge it reads all move together.

Pipeline Integration #

Alert rules are where the whole observability chain either pays off or fails quietly, and the failure is always the same shape: an alert that cannot fire because its input is missing. The ExporterDown rule exists precisely to break that chain — when the exporter dies, up == 0 pages within its for window, so the on-call engineer learns the monitoring is blind rather than trusting stale-green panels. Pair it with the heartbeat panel built in building Grafana dashboards for replication lag; the dashboard shows the gap and the rule pages on it, driven by the same up series.

For teams whose consumers are Python ETL processes, resist encoding replication thresholds inside the consumer. A consumer that decides on its own when lag is dangerous duplicates logic that belongs in one place and cannot see slots other consumers own. Let Prometheus own the thresholds against the exporter’s gauges, and have the consumer emit only what it uniquely knows — its last applied LSN and retry count — so a rule can cross-check the consumer’s progress against the slot’s confirmed_flush_lsn. When the exporter’s flat query model cannot express a needed signal, the bespoke collector in async monitoring integration publishes to the same Prometheus, and these rules alert on its series identically. On failover, the LogicalSlotInactive and SlotWalStatusNotReserved rules are the first to fire if a promotion leaves a slot behind — the runbook response is a fresh subscription sync once the new primary’s slot state is confirmed.

Authoritative references #

• Prometheus: Alerting rules — the groups:/rules: schema, for, labels, and annotations.
• Prometheus: Recording and alerting with promtool — unit-testing rule groups before deploy.
• Alertmanager: Configuration — routing trees, group_by, and repeat_interval for severity-based paging.

Related #

• Deploying postgres_exporter for replication metrics — the gauges every expr on this page reads.
• Building Grafana dashboards for replication lag — the panels that visualize the same thresholds these rules page on.
• Monitoring & Alerting — the parent guide this alerting layer completes.