Django select_for_update within celery task and race condition and unique together

Question

The application has these relevant models, Company, Node, and Log.

class VCompany(models.Model):
    company_name = models.CharField(max_length=50,  ...)

class VNode(models.Model):
            company = models.ForeignKey(VCompany, ...)
            name1 = models.CharField(...)
            name2 = models.CharField(...)

class Log(models.Model):
    node = models.ForeignKey(VNode, ... )

VNode's must be unique. Unfortunately, there is no simple unique/unique together. The combination of company and name1, if name1 is known, is unique. The combination of company and name2, if name2 is known, is unique. Either name1 or name2 or both is known.

Question 1: Is it possible to construct a unique clause for the above case? How?

A celery task is used to create records in Logs by processing some data in files. There are a number of celery task running concurrently to create the log records. As log records are created it may be necessary to create a vnode. This leads to duplicate vnode records being created. To address this issue I'm doing this:

with transaction.atomic():
    # Get the company for update as a sync method.
    company = VCompany.objects.select_for_update().get(company_name=company)
    if name1:
        node, create = VNode.objects.get_or_create(company=company, name1=... )
        ... set other node data ...
        node.save()
        return node
    if name2:
        node, create = VNode.objects.get_or_create(company=company, name2=... )
        ... set other node data ...
        node.save()
        return node

This code apparently works. However, if I remove the with transaction.atomic() it reliably fails.

Question 2: Why is transaction.atomic() needed? Is it needed?

Question 3: When does django's ORM release the select_for_update? Is get_or_create causing it to release before the new node is created such that a race condition occurs?

Thanks for any clues!

Answer 1

Question 1: I don't know

Question 2:

transaction_atomic is needed because select_for_update only holds the lock for the duration of the current transaction. Otherwise, (if you aren't in any other transaction) it will release the lock immediately.

Question 3:

The lock is released when the current transaction closes (commit or roll back).

get_or_create isn't atomic and will only be thread safe if the db has the correct uniqueness constraints. If get_or_create violates a uniqueness constraint it will raise an IntegrityError.

In your case, locking on VCompany should make it impossible for the get_or_create method on VNode to fail. However, that's only true if you correctly lock everywhere you try to create VNode objects.