Multiple concurrent database transactions with Django?

Question

Is it possible to use multiple concurrent transactions within one Django process?

Specifically, I've got two functions which should each execute concurrently (yielding between each other), but should each be in their own transaction. For example:

def process_things(callback, things):
    with start_transaction():
        for thing in things:
            obj = Thing.objects.create(…)
            callback(obj)

def callback(obj):
    with start_separate_transaction():
        …
        ThingProcessingLog.objects.create(…)

I'd like each function to be operating in an independent database transaction, so in this example, the ThingProcessingLog records will be created and visible immediately, but the Thing objects won't be visible until they have all been processed.

How can I do this with Django's ORM?

Answer 1

I did a little test (but with no transaction but I think it might work)

I setup 2 dbs in the django's settings:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'mydb',
        'USER': 'mydb',
        'PASSWORD': 'mydb',
        'HOST': '127.0.0.1',
        'PORT': '3306',
    },
    'alias': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'mydb',
        'USER': 'mydb',
        'PASSWORD': 'mydb',
        'HOST': '127.0.0.1',
        'PORT': '3306',
    }
}

Then in a django shell I did that:

Table1.objects.using('default').all()
Table1.objects.using('alias').all()

Table2.objects.using('default').all()
Table2.objects.using('alias').all()

In the same time I did this in mysql

mysql> show processlist;
+------+------+-----------------+------+---------+------+-------+------------------+
| Id   | User |     Host        | db   | Command | Time | State | Info             |
+------+------+-----------------+------+---------+------+-------+------------------+
| 1314 | mydb | localhost:40224 | mydb | Query   |    0 | NULL  | show processlist |
| 1315 | mydb | localhost:40225 | mydb | Sleep   |    5 |       | NULL             |
| 1316 | mydb | localhost:40226 | mydb | Sleep   |    5 |       | NULL             |
+------+------+-----------------+------+---------+------+-------+------------------+

So I suppose you will have a transaction for each db (which are actually the same db).

Answer 2

You might need to look at 'savepoint' in the django'sdoc