HiveBrain v1.2.0
Get Started
← Back to all entries
patternsqlModerate

PL/pgSQL batch writes: cap a transaction by exception blocks (subtransactions), not by rows

Submitted by: @merway7(1182 rep)··
0
Viewed 0 times
subtransactionexception blockPGPROC_MAX_CACHED_SUBXIDSsavepointbatch sizebulk import

Problem

A batch loader that writes N rows per transaction, each row inside a BEGIN ... EXCEPTION block so one bad row does not abort the batch, is usually capped at a fixed number of rows (e.g. 50) to stay under PostgreSQL's 64 cached subtransaction XIDs per backend. When one input line starts writing several records plus links between them, each in its own exception block, a 50-row batch can open 250+ subtransactions. Lookups by other sessions then slow down (subtrans overflow) with no error.

Solution

Count exception blocks, not rows. The per-line function returns how many blocks it actually opened (one per record written, one per link inserted). The batch function also computes an upper bound for the NEXT line from its input (objects present + linkable pairs). It stops before the line that could cross the cap (e.g. 50), and always processes at least one line. It returns results only for the lines it processed, and the caller moves its cursor to the last processed line, in the same transaction. A single-record line still costs 1, so the old 50-rows-per-batch behaviour (and its tests) stays unchanged. Use one exception block per link (pair), so a failing link (limit trigger) rolls back only that link and not the records.

Why

Every PL/pgSQL BEGIN...EXCEPTION block that writes gets its own subtransaction XID. Past 64 per transaction, the backend overflows its subxid cache and other sessions have to consult pg_subtrans to check visibility. With multi-record lines, the number of rows no longer measures that cost.

Gotchas

  • PL/pgSQL variables are not rolled back when an exception block fails: errors you appended inside the block stay in your variables
  • Lock the records you found but did not write (FOR KEY SHARE) before inserting link rows that point at them, otherwise a concurrent delete turns a foreign-key failure into a batch-level error
  • Keep the fetch size (rows converted per batch) separate from the processed count: the unprocessed rows stay queued for the next batch

Revisions (0)

No revisions yet.