best practice to override saving method in django model to be async using celery

Question

i am building a cloud system, i have two apps, server apps which includes full functionality, and client app which including only input method, so i am installing the client app in the customer branch as local app,

i want override any models in the apps after saving the models locally, i will call a celery task to add this model in the queue to make sure it will arrive, even if the internet is down, i will do retry until internet is getting up,

now i want the best practice to have a generic way to do it to any model

i have two option

1- overriding save method like this

def save(self, *args, **kwargs):
    super(Model, self).save(*args, **kwargs)
    save_task.delay(self)

or using signals like this

post_save.connect(save-task.delay, sender=Model)

which one is the best practice and i can make it generics for all the models of this project?

Answer 1

.save() is just a bunch of signals executed one after the other. here's a shortened version of the process from the documentation:

1. Emit a pre-save signal. [...]

2. Pre-process the data. [...] Most fields do no pre-processing [...] only used on fields that have special behavior [...] documentation doesn’t yet include a list of all the fields with this “special behavior.”

3. Prepare the data for the database. Each field is asked to provide its current value in a data type that can be written to the database. Most fields require no data preparation [...] integers and strings are ‘ready to write’ as a Python object [...] complex data types often require some modification. [...]

4. Insert the data into the database. [...]

5. Emit a post-save signal. [...]

In your case, you're not doing anything in the middle of that process. You only need to do it after the model has already been saved. So there's no need to use signals.

Now what you're actually asking, is how to make sure a task will be executed eventually. Well:

1. I'm pretty sure you can solve this using celery
2. You should hook up the applications to a single db (if you can), don't save things locally and then update a server, that could turn ugly.

but, if you truly think there's a fair chance of the internet going down or anything like that, and you're sure there's no better way to link your apps, I would suggest you add a new model that keeps track of what's been updated. Something like this:

class Track(models.Model):
    modelname = models.CharField(max_length=20)
    f_pk = models.IntegerField()
    sent = models.BooleanField()

    def get_obj(self):
        try:
            # we want to do modelname.objects.get(pk=self.f_pk), so:
            return getattr( getattr(self.modelname, 'objects'), 'get')(pk=self.f_pk)
        except:
            return False

Notice how I'm not linking it to a certain model but rather giving it tools to fetch on any model you damn well please. Then, for each model you want to keep track of, you add this:

class myModel(models.Model):
    ...
    def save(self, *args, **kwargs):
        super(Model, self).save(*args, **kwargs)
        t = Track(modelname=self.__class__.__name__, f_pk=self.pk, sent=False)
        t.save()

Then scheduale a task that will Track objects with sent=False and try to save them:

unsent = Track.objects.filter(sent=False)
for t in unsent:
     obj = t.get_obj()
     # check if this object exists on the server too
     # if so:
         t.sent = True
         t.save()

p.s.

remember how I mentioned things could get ugly? it's been moments since I posted this, and I already see how. Notice how I use a pk and modelname to figure out if a model is saved in both places, right? but, pk's are (by default in django) an auto-incremented field. If the application runs in two places, or even if you run it locally and something wrong happens once, than the pks ca quickly be out of sync.

Say I saved once object, it gets a pk of 1 on both local and server.

local             server
name    pk   ++   name    pk
obj1    1    ++   obj1    1

Then I save another one but the internet went down.

local             server
name    pk   ++   name    pk
obj1    1    ++   obj1    1
obj2    2    ++

Next time it's up, I add a new object, but this happens before the schedualed task runs. So now my local db has 3 objects, and my server has 2, and those have different pk's, get it?

local             server
name    pk   ++   name    pk
obj1    1    ++   obj1    1
obj2    2    ++   obj3    2
obj3    3    ++

and after the schedualed task will run we'll have this:

local             server
name    pk   ++   name    pk
obj1    1    ++   obj1    1
obj2    2    ++   obj3    2
obj3    3    ++   obj2    3

See how easily this can get out of hand? To fix this, each tracked model will have to have some sort of unique identifier, and you'll need to somehow tell the Track model how to follow it. It's a headache. Better not to save things locally but to link everything together