Building Grafana Dashboards for Replication Lag

A Grafana dashboard for replication lag earns its place only when it renders retained WAL as a fraction of maxslotwalkeepsize, breaks lag out per slot with a…

A Grafana dashboard for replication lag earns its place only when it renders retained WAL as a fraction of max_slot_wal_keep_size, breaks lag out per slot with a template variable, and treats a dead exporter as unknown rather than a flat healthy line — the panel-design decisions this guide details on top of the metrics the Monitoring & Alerting pipeline exports. A dashboard that plots raw byte counts without the cap as a reference, or that shows the last-scraped value forever after the exporter dies, is the visualization equivalent of no monitoring at all.

This guide is about panel construction specifically: the PromQL behind each panel, per-slot template variables, threshold coloring that maps to the alert boundaries, converting a byte count into a fraction of the retention cap, and a heartbeat panel that makes exporter absence visible. It assumes the postgres_exporter gauges already exist and Prometheus is scraping them.

Replication-lag dashboard layout and threshold bands A slot template variable at the top of the dashboard drives every panel. Below it, four panels: a fraction-of-cap gauge with green under fifty percent, amber to eighty percent, and red above; a per-slot apply-lag time series; an active-state panel that reads red when a slot is inactive; and a heartbeat panel that goes unknown when the exporter stops reporting. Variable: $slot = orders_cdc ▾ drives every panel below via label_values(slot_name) retained / cap 0–50% ok 50–80% warn 80–100% crit fraction, not raw bytes apply lag / slot thr active DOWN value → color exporter heartbeat UNKNOWN no data ≠ healthy
One slot variable drives every panel; the fraction-of-cap gauge, per-slot lag series, active-state panel, and heartbeat make disk risk, latency, liveness, and exporter health readable at a glance.

Panel Semantics

Each panel answers one operational question, and the PromQL, unit, and coloring should be chosen so the answer is unambiguous. The table fixes what each panel plots and where its color boundaries sit, so the dashboard and the alert rules that shadow it agree on when a number is bad.

Panel PromQL core Unit Color boundary
Retained fraction retained_bytes / slot_wal_keep_bytes percent (0–1) green < 0.5, amber 0.5–0.8, red > 0.8
Apply lag pg_repl_slot_lag_unconfirmed_bytes{slot_name="$slot"} bytes red above 256 MB
Sender flush lag pg_repl_sender_flush_lag_seconds seconds red above 60 s
Active state pg_repl_slot_lag_active{slot_name="$slot"} 0/1 mapping red when value = 0
Heartbeat up{job="postgres"} 0/1 mapping unknown/no-data when absent

The retained-fraction panel is the one that prevents an outage, and it must be a fraction, not a byte count. A raw retained_bytes graph forces the viewer to remember the cap and do the division in their head; dividing by the exported slot_wal_keep_bytes gauge produces a self-scaling 0–1 value whose thresholds never drift when someone changes max_slot_wal_keep_size. The byte arithmetic those gauges come from — restart_lsn and confirmed_flush_lsn against the write head — is detailed in WAL stream mechanics.

The active-state panel deserves a value mapping rather than a bare number. A stat panel showing 0 or 1 is unreadable under pressure; map 1 to a green “UP” and 0 to a red “DOWN” so the state is legible at a glance from across a room. The same applies to the heartbeat: up is a 0/1 series, but its meaning is liveness, and a value mapping plus a “No data” state is what turns a numeric series into an operational signal. Choose the time range and refresh so the dashboard’s resolution matches the alert cadence — a board on a 5-minute range with a 1-hour refresh will lag the pager by design and mislead whoever opens it during an incident.

Diagnostic Patterns

Build the retained-fraction panel as a time series or gauge with three threshold steps. The query divides pinned WAL by the exported cap; guard against a divide-by-zero if the cap is briefly unavailable.

code
# Retained WAL as a fraction of the configured cap, per slot.
# Threshold steps: 0 green, 0.5 amber (yellow), 0.8 red.
pg_repl_slot_lag_retained_bytes{slot_name=~"$slot"}
  / clamp_min(pg_repl_slot_wal_keep_bytes, 1)

The apply-lag time series should overlay every selected slot so a single lagging consumer stands out against its healthy peers. Use the regex form of the template variable so an “All” selection expands to every slot.

code
# Per-slot unconfirmed backlog; one line per slot when $slot = All.
pg_repl_slot_lag_unconfirmed_bytes{slot_name=~"$slot"}

The heartbeat panel is what stops a dead exporter from reading as healthy. Plot up for the postgres job, and configure the panel so a null renders as “No data” mapped to a neutral or warning color — never let Grafana connect across the gap and imply continuity that is not there.

code
# Exporter liveness. In the panel: "Connect null values = Never",
# and a value mapping so absent/0 shows as UNKNOWN, not a flat green line.
up{job="postgres"}

Define the slot variable once so every panel inherits it. Point it at label_values over any of the slot gauges.

code
# Dashboard variable $slot (type: Query, data source: Prometheus)
label_values(pg_repl_slot_lag_active, slot_name)
# Enable "Include All option" with the All value set to .* for the regex matchers above.

Add a deploy-annotation query so schema migrations and connector restarts appear as vertical markers on the lag panels. Correlating a lag spike with a deploy is the difference between diagnosing a cause in seconds and chasing it for an hour; a slot that goes inactive at the exact moment of a connector redeploy is a config regression, not a database fault.

code
# Annotation query (Prometheus data source): mark exporter restarts on the timeline.
changes(process_start_time_seconds{job="postgres"}[$__interval]) > 0

Safe Deployment Sequence

Ship the dashboard so it is reproducible and never edited only in the UI.

1. Prototype in the UI, but treat it as throwaway. Build one panel at a time against live data, confirm the PromQL returns the series you expect, and check the coloring flips at the intended boundary by temporarily lowering the threshold.

2. Export the model to JSON and commit it. Grafana dashboards are JSON; keep the source of truth in version control, not in a database only an admin can reach. The panel skeleton for the retained-fraction gauge looks like this:

json
{
  "title": "Retained WAL / cap",
  "type": "gauge",
  "fieldConfig": {
    "defaults": {
      "unit": "percentunit",
      "min": 0, "max": 1,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "green",  "value": null},
          {"color": "yellow", "value": 0.5},
          {"color": "red",    "value": 0.8}
        ]
      }
    }
  },
  "targets": [
    {"expr": "pg_repl_slot_lag_retained_bytes{slot_name=~\"$slot\"} / clamp_min(pg_repl_slot_wal_keep_bytes, 1)"}
  ]
}

3. Provision it declaratively. Load the JSON through a Grafana dashboard provisioning file so a rebuilt Grafana comes up with the panel already present, and templating variables resolve against the live Prometheus.

yaml
# /etc/grafana/provisioning/dashboards/replication.yaml
apiVersion: 1
providers:
  - name: replication
    folder: Replication
    type: file
    options:
      path: /var/lib/grafana/dashboards/replication.json

Revert is a git revert plus a provisioning reload; because dashboards are read-only observers, rolling one back never touches replication. Keep the JSON and the alert rules in the same change so a threshold and the panel that visualizes it move together.

Pipeline Integration

A dashboard is only as trustworthy as its treatment of missing data, and this is where most replication dashboards quietly fail. When the exporter dies, its gauges stop updating; a naively configured time-series panel will hold the last value and a stat panel will keep showing green. Set every panel’s null handling to leave gaps rather than connect them, and add the heartbeat panel so absence is a visible state. This mirrors the alerting side, where an up == 0 rule pages on the same condition — the dashboard shows it and the rule fires on it, built from the same series authored in writing alert rules for slot and WAL thresholds.

For teams that also run the bespoke asyncpg collector from async monitoring integration, join its series onto the same panels by slot name so a consumer’s self-reported last-applied LSN sits beside the exporter’s view of the slot. The two should agree; a persistent gap between the consumer’s committed offset and the slot’s confirmed_flush_lsn is itself a signal, and having both on one graph is how an on-call engineer localizes a stall to the database or the consumer. Keep the slot variable driving both, and template the data source too if you run one Grafana across several database clusters so the same dashboard serves every environment.

Size the retention window on the underlying Prometheus to the time span the dashboard shows. A board that offers a 7-day view over a Prometheus retaining only 24 hours renders empty history and invites the wrong conclusion that replication was fine when in fact there is simply no data. Match them explicitly, and prefer a longer retention for the retained-fraction series specifically, because a slow leak — a slot that gains a few percent of the cap per day — is only visible over days, not minutes, and is exactly the failure a real-time-only view hides until it is critical.

Authoritative references