snippetsqlMinor
How to keep an unique counter per row with PostgreSQL?
Viewed 0 times
postgresqluniqueperwithkeephowrowcounter
Problem
I need to keep an unique (per-row) revision number in a document_revisions table, where the revision number is scoped to a document, so it's not unique to the whole table, only to the related document.
I initially came up with something like:
But there is a race condition!
I'm trying to solve it with
Is the following acceptable, or am I doing it wrong, or is there a better solution?
Shouldn't I lock the document row (key1) for a given operation (key2) instead? So that would be the proper solution:
Maybe I'm not used to PostgreSQL and a SERIAL can be scoped, or maybe a sequence and
I initially came up with something like:
current_rev = SELECT MAX(rev) FROM document_revisions WHERE document_id = 123;
INSERT INTO document_revisions(rev) VALUES(current_rev + 1);But there is a race condition!
I'm trying to solve it with
pg_advisory_lock, but the documentation is a bit scarce and I don't fully understand it, and I don't want to lock something by mistake.Is the following acceptable, or am I doing it wrong, or is there a better solution?
SELECT pg_advisory_lock(123);
current_rev = SELECT MAX(rev) FROM document_revisions WHERE document_id = 123;
INSERT INTO document_revisions(rev) VALUES(current_rev + 1);
SELECT pg_advisory_unlock(123);Shouldn't I lock the document row (key1) for a given operation (key2) instead? So that would be the proper solution:
SELECT pg_advisory_lock(id, 1) FROM documents WHERE id = 123;
current_rev = SELECT MAX(rev) FROM document_revisions WHERE document_id = 123;
INSERT INTO document_revisions(rev) VALUES(current_rev + 1);
SELECT pg_advisory_unlock(id, 1) FROM documents WHERE id = 123;Maybe I'm not used to PostgreSQL and a SERIAL can be scoped, or maybe a sequence and
nextval() would do the job better?Solution
(I came to this question when trying to re-discover an article about this topic. Now that I've found it, I'm posting it here in case others are in pursuit of an alternate option to the currently chosen answer—windowing with
I have this same use case. For each record inserted into a specific project in our SaaS we need a unique, incrementing number which can be generated in the face of concurrent
This article describes a nice solution, which I'll summarise here for ease and posterity.
Alternatively, you might be able to use a CTE to do this at the application layer (though I prefer it to be a trigger for consistency's sake):
This is similar in principle to how you were trying to solve it initially, except that by modifying a counter row in a single statement it blocks reads of the stale value until the
Here's a transcript from
As you can see, you have to be careful about how
That makes
row_number())I have this same use case. For each record inserted into a specific project in our SaaS we need a unique, incrementing number which can be generated in the face of concurrent
INSERTs and is ideally gapless.This article describes a nice solution, which I'll summarise here for ease and posterity.
- Have a separate table which acts as the counter to provide the next value. It will have two columns,
document_idandcounter.counterwill beDEFAULT 0Alternatively, if you already have adocumententity that groups all versions, acountercould be added there.
- Add a
BEFORE INSERTtrigger to thedocument_versionstable which atomically increments the counter (UPDATE document_revision_counters SET counter = counter + 1 WHERE document_id = ? RETURNING counter) and then setsNEW.versionto that counter value.
Alternatively, you might be able to use a CTE to do this at the application layer (though I prefer it to be a trigger for consistency's sake):
WITH version AS (
UPDATE document_revision_counters
SET counter = counter + 1
WHERE document_id = 1
RETURNING counter
)
INSERT
INTO document_revisions (document_id, rev, other_data)
SELECT 1, version.counter, 'some other data'
FROM "version";This is similar in principle to how you were trying to solve it initially, except that by modifying a counter row in a single statement it blocks reads of the stale value until the
INSERT is committed.Here's a transcript from
psql showing this in action:scratch=# CREATE TABLE document_revisions (document_id integer, rev integer, other_data text, PRIMARY KEY (document_id, rev));
CREATE TABLE
scratch=# CREATE TABLE document_revision_counters (document_id integer PRIMARY KEY, counter integer DEFAULT 0);
CREATE TABLE
scratch=# WITH version AS (
INSERT INTO document_revision_counters (document_id) VALUES (2)
ON CONFLICT (document_id)
DO UPDATE SET counter = document_revision_counters.counter + 1
RETURNING counter;
)
INSERT
INTO document_revisions (document_id, rev, other_data)
SELECT 2, version.counter, 'doc 1 v1'
FROM "version";
INSERT 0 1
scratch=# WITH version AS (
INSERT INTO document_revision_counters (document_id) VALUES (2)
ON CONFLICT (document_id)
DO UPDATE SET counter = document_revision_counters.counter + 1
RETURNING counter;
)
INSERT
INTO document_revisions (document_id, rev, other_data)
SELECT 2, version.counter, 'doc 1 v2'
FROM "version";
INSERT 0 1
scratch=# WITH version AS (
INSERT INTO document_revision_counters (document_id) VALUES (2)
ON CONFLICT (document_id)
DO UPDATE SET counter = document_revision_counters.counter + 1
RETURNING counter;
)
INSERT
INTO document_revisions (document_id, rev, other_data)
SELECT 2, version.counter, 'doc 2 v1'
FROM "version";
INSERT 0 1
scratch=# SELECT * FROM document_revisions;
document_id | rev | other_data
-------------+-----+------------
2 | 1 | doc 1 v1
2 | 2 | doc 1 v2
2 | 1 | doc 2 v1
(3 rows)As you can see, you have to be careful about how
INSERTs happen, hence the trigger version, which looks like this:CREATE OR REPLACE FUNCTION set_doc_revision()
RETURNS TRIGGER AS $ BEGIN
WITH version AS (
INSERT INTO document_revision_counters (document_id, counter) VALUES (NEW.document_id, 1)
ON CONFLICT (document_id)
DO UPDATE SET counter = document_revision_counters.counter + 1
RETURNING counter
)
SELECT INTO NEW.rev counter FROM version; RETURN NEW; END;
$ LANGUAGE 'plpgsql';
CREATE TRIGGER set_doc_revision BEFORE INSERT ON document_revisions
FOR EACH ROW EXECUTE PROCEDURE set_doc_revision();That makes
INSERTs much more straight forward and the integrity of the data more robust in face of INSERTs originating from arbitrary sources:scratch=# INSERT INTO document_revisions (document_id, other_data) VALUES (1, 'baz');
INSERT 0 1
scratch=# INSERT INTO document_revisions (document_id, other_data) VALUES (1, 'foo');
INSERT 0 1
scratch=# INSERT INTO document_revisions (document_id, other_data) VALUES (1, 'bar');
INSERT 0 1
scratch=# INSERT INTO document_revisions (document_id, other_data) VALUES (42, 'meaning of life');
INSERT 0 1
scratch=# SELECT * FROM document_revisions;
document_id | rev | other_data
-------------+-----+-----------------
1 | 1 | baz
1 | 2 | foo
1 | 3 | bar
42 | 1 | meaning of life
(4 rows)Code Snippets
WITH version AS (
UPDATE document_revision_counters
SET counter = counter + 1
WHERE document_id = 1
RETURNING counter
)
INSERT
INTO document_revisions (document_id, rev, other_data)
SELECT 1, version.counter, 'some other data'
FROM "version";scratch=# CREATE TABLE document_revisions (document_id integer, rev integer, other_data text, PRIMARY KEY (document_id, rev));
CREATE TABLE
scratch=# CREATE TABLE document_revision_counters (document_id integer PRIMARY KEY, counter integer DEFAULT 0);
CREATE TABLE
scratch=# WITH version AS (
INSERT INTO document_revision_counters (document_id) VALUES (2)
ON CONFLICT (document_id)
DO UPDATE SET counter = document_revision_counters.counter + 1
RETURNING counter;
)
INSERT
INTO document_revisions (document_id, rev, other_data)
SELECT 2, version.counter, 'doc 1 v1'
FROM "version";
INSERT 0 1
scratch=# WITH version AS (
INSERT INTO document_revision_counters (document_id) VALUES (2)
ON CONFLICT (document_id)
DO UPDATE SET counter = document_revision_counters.counter + 1
RETURNING counter;
)
INSERT
INTO document_revisions (document_id, rev, other_data)
SELECT 2, version.counter, 'doc 1 v2'
FROM "version";
INSERT 0 1
scratch=# WITH version AS (
INSERT INTO document_revision_counters (document_id) VALUES (2)
ON CONFLICT (document_id)
DO UPDATE SET counter = document_revision_counters.counter + 1
RETURNING counter;
)
INSERT
INTO document_revisions (document_id, rev, other_data)
SELECT 2, version.counter, 'doc 2 v1'
FROM "version";
INSERT 0 1
scratch=# SELECT * FROM document_revisions;
document_id | rev | other_data
-------------+-----+------------
2 | 1 | doc 1 v1
2 | 2 | doc 1 v2
2 | 1 | doc 2 v1
(3 rows)CREATE OR REPLACE FUNCTION set_doc_revision()
RETURNS TRIGGER AS $$ BEGIN
WITH version AS (
INSERT INTO document_revision_counters (document_id, counter) VALUES (NEW.document_id, 1)
ON CONFLICT (document_id)
DO UPDATE SET counter = document_revision_counters.counter + 1
RETURNING counter
)
SELECT INTO NEW.rev counter FROM version; RETURN NEW; END;
$$ LANGUAGE 'plpgsql';
CREATE TRIGGER set_doc_revision BEFORE INSERT ON document_revisions
FOR EACH ROW EXECUTE PROCEDURE set_doc_revision();scratch=# INSERT INTO document_revisions (document_id, other_data) VALUES (1, 'baz');
INSERT 0 1
scratch=# INSERT INTO document_revisions (document_id, other_data) VALUES (1, 'foo');
INSERT 0 1
scratch=# INSERT INTO document_revisions (document_id, other_data) VALUES (1, 'bar');
INSERT 0 1
scratch=# INSERT INTO document_revisions (document_id, other_data) VALUES (42, 'meaning of life');
INSERT 0 1
scratch=# SELECT * FROM document_revisions;
document_id | rev | other_data
-------------+-----+-----------------
1 | 1 | baz
1 | 2 | foo
1 | 3 | bar
42 | 1 | meaning of life
(4 rows)Context
StackExchange Database Administrators Q#47774, answer score: 6
Revisions (0)
No revisions yet.