Selecting Avro Codecs for CDC Payloads

Choosing an Avro compression codec for CDC payloads is a CPU-versus-size trade-off that, at change-stream throughput, decides whether you burn broker storage or serializer cores. This page sits under the JSON to Avro transformation guide and isolates one decision: which codec to apply to Avro object-container files bound for a lake or batch sink, which compression.type to set on the Kafka producer that carries the live stream, and how the two interact with batch size and TOAST-heavy rows. All measurements assume PostgreSQL 14–17 feeding a Debezium 2.x connector, and the numbers below are representative ratios you should re-measure against your own row shapes.

Compression on a CDC path is not the same problem as compressing a static dataset. The payloads are small (a single row image), they arrive continuously, and they are latency-sensitive because a downstream consumer is waiting on each batch. A codec that wins on a 1 GB benchmark file can lose badly on a stream of 300-byte change events, because per-message framing overhead and CPU-per-event dominate when the unit of work is tiny. Pick the wrong codec and the failure is quiet but expensive: either the broker disk and cross-AZ egress bill climb because you shipped near-raw bytes, or the serializer saturates CPU and the replication slot starts retaining WAL because decode cannot keep pace with the write rate.

Codec Selection Semantics #

There are two distinct compression layers on a CDC path and they are configured in different places. The Avro object-container codec (null, deflate, snappy, zstd, bzip2) applies when you write Avro to files — a lake sink, an archival dump, a batch export — and is embedded in the container header. The Kafka batch compression (compression.type = none, gzip, snappy, lz4, zstd) applies to the live stream and compresses whole producer batches, not individual Confluent-framed records. The table below is the Avro-codec decision; the Kafka mapping is called out in the last column because the reasoning transfers directly.

The ratio numbers move with row shape. Wide rows with repeated JSON-like text or many null union branches compress far better than narrow numeric rows, and a TOAST-heavy table carrying large text/jsonb values will see zstd reach 4x or more while a table of bigint keys and timestamps barely clears 1.5x. Measure per topic; do not carry a single codec assumption across a heterogeneous set of tables. Batch size is the multiplier: compression works on the batch, so a linger.ms of 10–20 ms that groups tens of small change events lets the codec find cross-record redundancy that per-event compression never sees — a raised linger.ms often improves the effective ratio more than switching codecs does.

Diagnostic Patterns #

Compression only matters if the stream is flowing, so start at the slot. A serializer pinned at 100% CPU on an over-aggressive codec stops acknowledging WAL, and the symptom shows up first as slot lag on the publisher.

-- Publisher: is codec CPU cost starving the consumer and pinning WAL?
SELECT slot_name,
       active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))         AS retained_wal,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unconfirmed
FROM pg_replication_slots
WHERE slot_type = 'logical'
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;
-- retained_wal climbing while the consumer host shows high CPU => codec too heavy for the write rate.

Measure the effective ratio empirically rather than trusting the benchmark table — write a fixed sample through each codec and compare on-wire bytes.

import io, fastavro

def encoded_size(records, schema, codec):
    buf = io.BytesIO()
    fastavro.writer(buf, schema, records, codec=codec)   # 'null','deflate','snappy','zstandard','bzip2'
    return buf.tell()

raw = encoded_size(sample, schema, "null")
for codec in ("snappy", "zstandard", "deflate", "bzip2"):
    size = encoded_size(sample, schema, codec)
    print(f"{codec:10s} ratio={raw/size:4.2f}x  bytes={size}")

On the Kafka side, confirm the broker sees compressed batches and watch the ratio in the producer metrics.

# Producer JMX: compression-rate-avg is the fraction of uncompressed size actually sent.
# A value near 1.0 means compression is effectively off — check compression.type is set on the PRODUCER.
kafka-run-class kafka.tools.JmxTool \
  --object-name 'kafka.producer:type=producer-metrics,client-id=*' \
  --attributes compression-rate-avg,batch-size-avg,record-size-avg

• compression-rate-avg near 1.0 — batches are not compressing; the codec is unset on the producer or batches are single-record (raise linger.ms/batch.size).
• Consumer host CPU sustained > 80% with rising slot lag — the codec (or its level) is too heavy for the throughput; drop from deflate/high-level zstd to snappy or zstd level 3.
• Effective ratio < 1.3x on a wide table — the row is already near-incompressible (encrypted or binary columns); do not pay codec CPU for no gain.

Safe Deployment Sequence #

Changing a codec is not free to roll out because consumers must be able to read both the old and new encoding during the transition. Sequence it so no consumer meets an encoding it cannot decode.

1. Baseline. Record current on-wire bytes/event, compression-rate-avg, and consumer-host CPU. Without the baseline you cannot prove the change paid off.

2. Confirm consumer support. Verify every consumer library can decode the target codec — snappy and zstd need the native codec libs installed (python-snappy, zstandard). A consumer missing the lib fails at decode, not at deploy.

3. Change the producer, not the topic history. For the live stream, set compression.type on the producer; existing messages keep their original codec and remain readable because compression is per-batch and self-describing.

producer = Producer({
    "bootstrap.servers": "kafka:9092",
    "compression.type": "zstd",     # was snappy
    "linger.ms": 20,                # batch window: raises effective ratio on small events
    "batch.size": 65536,            # let batches fill before send
    "enable.idempotence": True,
})

4. Watch, then revert if needed. Track compression-rate-avg, consumer CPU, and the slot lag query above for at least one peak window. The revert is a single producer config change back to the prior compression.type — no topic rewrite, because consumers decode each batch by its own header.

For Avro file sinks the equivalent revert is to point new files at the old codec; already-written container files remain readable because the codec is stamped in each file’s header.

Pipeline Integration #

The codec decision lives at the seam between the serializer and the broker, so it interacts with everything the Debezium connector configuration already sets — max.batch.size, linger.ms, and the converter that produces the Avro bytes. Keep these patterns in mind when tuning:

• Batch to compress, but bound the batch. A larger linger.ms/batch.size improves ratio but adds latency and enlarges the in-flight buffer; hold linger.ms at 10–20 ms for CDC so ratio improves without the stream feeling stalled. Never let batching grow unbounded to chase ratio.
• TOAST-heavy tables win most. Rows carrying large text/jsonb from out-of-line storage are where zstd earns its keep; route those tables to zstd and leave narrow high-frequency key/timestamp tables on snappy where the ratio gain does not justify the CPU.
• Idempotent, LSN-guarded apply. Codec choice never changes delivery semantics — the stream is still at-least-once, so consumers must upsert on the primary key and drop any event whose source LSN is not newer than the stored high-water mark, exactly as they would under any codec.
• Failover replays are re-compressed transparently. After a primary failover the connector may replay a window of events; each replayed batch is compressed by the current producer setting and decoded by the consumer from its batch header, so a codec change made mid-incident does not corrupt the replay.

The registry contract that governs the schema inside these compressed payloads is a separate concern handled in integrating the Confluent Schema Registry, and the partitioning and dead-letter routing of the compressed records is covered in event routing and Kafka integration.

Authoritative references #

• Apache Avro specification: object container files and codecs — the canonical list of container codecs (null, deflate, snappy, zstandard, bzip2) and how each is stamped in the file header.
• Apache Kafka: producer compression.type — batch-level compression options for the live stream and their trade-offs.
• Debezium PostgreSQL connector properties — max.batch.size and converter settings that determine the batch the codec compresses.

Related #

• JSON to Avro Transformation — the parent workflow; this page tunes the compression of the records that transform emits.
• Integrating the Confluent Schema Registry — the schema contract carried inside these compressed payloads.
• Debezium Connector Configuration — the batch and converter parameters that set the unit of work the codec compresses.