PostgreSQL sometimes fails to detect serialization failures

Question

I have an http-server in application language that looks roughly like this:

item = SELECT item FROM table WHERE field = 'value'
if (item) {
  UPDATE another field on item
} else {
  INSERT item VALUE field = 'value' and so on
}

The field has unique constraint. About once every several minutes two requests with the same value come in at the same time. They both execute this same code so of course there is a race condition. I decided that if I wrap it in a transaction with SERIALIZABLE isolation level and retry on every serialization failure then the problem is solved. And on localhost it worked fine. I indeed see serialization failures thrown with error code 40001 as expected.

However on production in a database under load for some reason a Unique constraint violation is sporadically thrown with a code of 23505. It only happens sometimes. I put a lot of effort to reproduce it on localhost but failed. Everytime I get a normal 40001. I tried to run server code in separate processes and to put delays at different points to enforce certain orders of execution. It only happens on production. So I added extensive logging to prod. The logs say that every time when a problem occurs the item is not found at first but when INSERT happens it triggers Unique constraint violation. So it looks just ad a race condition.

But why sometimes under some conditions Postgres SERIALIZABLE transactions don't detect it as they should? Or how to debug it? It happens like several times in a day out of several dozens of cases.

The (almost) exact SQL look like this: (I omitted detailed fields lists and exact parameters)

START TRANSACTION
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE

SELECT *
FROM products
WHERE product_code = $1
LIMIT 1 -- product_code is a parameter

SELECT *
FROM licenses
WHERE license_key = $1
LIMIT 1 -- license_key is another and it's unique across the table
-- At this point SELECT did not return anything but it was inserted before this INSERT by exactly the same transaction

INSERT INTO licenses ("license_key", "product_id", "other_data",...)
VALUES ($1, $2, $3,...) RETURNING "id"
query failed: error: duplicate key value violates unique constraint "licenses_license_key_key"

Answer 1

Doing this in a serialized transaction should work, unless there is a bug.

As a workaround, use this command, which tries to add a new license; if the license key already exists, it adds the new hours to the existing value:

INSERT INTO licenses(license_key, product_id, hours)
VALUES ($1,
        (SELECT product_id FROM products WHERE product_code = $2 LIMIT 1),
        $3)
ON CONFLICT (license_key) DO UPDATE
SET hours = hours + EXCLUDED.hours
RETURNING id;

^{documentation}

Answer 2

This comment/answer was sent to me by Thomas Munro:

My guess is that they might have multiple unique indexes, and might have run into this problem.

Summary: It used to be that we always reported unique constraint violations (UCV) before detecting SSI failures, with the idea being that errors don't matter; SSI only makes promises about the transactions that commit. Then we added a one line tweak that makes it try to detect an SSI failure just before it raises a UCV (think of it as an error priority: IMHO SSI errors need higher priority because automatic transaction retry systems need reliable SSI errors; what are they supposed to do with UCV?). But now we know that that wasn't quite good enough. As described in that thread, before reporting UCV we probably need to run around and find all the indexes you'll insert into, to give them a chance to check for SSI errors, before we report UCV. In the case that was reported in that -bugs thread, you could change the outcome by changing the order that UNIQUE constraints were defined. Ugh. That'll require a bit of an architectural redesign, possibly including a new index AM entry function, but I think it's worth it (and if done right, it can also allow for SSI errors to have higher priority than exclusion constraint errors).