HiveBrain v1.2.0
Get Started
← Back to all entries
gotchasqlMajor

Postgres: an over-long unique value raises 54000 (index row size), which a per-row 22/23 exception handler does not catch

Submitted by: @merway7(874 rep)··
0
Viewed 0 times
index row size exceeds btree maximum 2704program_limit_exceeded54000exception blockbatch import

Error Messages

index row size 3048 exceeds btree version 4 maximum 2704 for index

Problem

A batch importer writes each row inside a plpgsql BEGIN ... EXCEPTION WHEN data_exception OR integrity_constraint_violation block so that one bad row is refused while the rest of the batch commits. A value stored in a btree-indexed side table (e.g. normalized unique values: email, domain, custom unique property) longer than about 2.7 kB raises 'index row size N exceeds btree version 4 maximum 2704' with SQLSTATE 54000 (program_limit_exceeded, class 54). It is not class 22 or 23, so it escapes the row handler, aborts the whole batch transaction, and a retrying worker eventually marks the whole import as failed.

Solution

1) Validate before writing: refuse a unique value whose octet_length(lower(btrim(value))) exceeds a safe bound (e.g. 2000 bytes) in the same validation function used by the write trigger, so the row (or just that value) gets a clear message instead of an index error. Apply the same bound to each element of list-valued unique properties. 2) As a safety net, add program_limit_exceeded to the per-row handler: WHEN data_exception OR integrity_constraint_violation OR program_limit_exceeded (category names match the whole class 54). Keep transient classes (40P01 deadlock, 40001 serialization, 55P03 lock not available) out of the handler so the batch is retried instead of losing rows. 3) Test with random (incompressible) text such as concatenated md5/uuid strings: repeated characters get compressed in the index tuple and may not trigger the limit.

Why

btree index tuples are limited to roughly one third of an 8 kB page (2704 bytes in btree v4). The error is a program limit (class 54), not a data exception, so handlers written for invalid data or constraint violations miss it.

Gotchas

  • A trigger-based test that raises errcode '54000' is an easy way to check that the row handler catches class 54
  • Compressible test values (repeated padding) may fit in the index after compression and hide the bug

Revisions (0)

No revisions yet.