Caching in Django's ManyToManyField

Question

I struggle with some caching issue inside Django. So far I've seen this issue only when running testsuite. The problem is that sometimes (it seems that this happens always on second invocation of the code), Django does not update it's cache or it becomes inconsistent.

The extracted code with some debugging is:

class Source(models.Model):
    name = models.CharField(max_length = 50)
    quality = models.IntegerField(default = 0)

class Reference(models.Model):
    url = models.URLField()
    source = models.ForeignKey(Source)

    class Meta:
        ordering = ['-source__quality']

class Issue(models.Model):
    references = models.ManyToManyField(Reference)
    master = models.ForeignKey(Reference, related_name = 'mastered_issue_set')

def auto_create(instance):
    issue = Issue.objects.create(master = instance)
    print issue.references.count(), issue.references.all()
    issue.references.add(instance)
    print issue.references.count(), issue.references.all()

At first invocation I correctly get following output:

0 []
1 [<Reference: test>]

However in second call to to auto_create, Django thinks there is one reference, but it does not give it to me:

0 []
1 []

This behavior of course breaks further code. Any idea what can be going wrong here or at least how to debug it?

PS: It looks like ordering on Reference class is causing this. But it is still unclear to me why.

Answer 1

In the end I've found what is causing this problem. It was my own caching code rather than Django's.

I had custom Source manager in place, which returned and cached some standard source:

class SourceManager(models.Manager):
    url_source = None
    def get_generic(self):
        if self.url_source is None:
            self.url_source, created = self.get_or_create(name = 'URL', quality = 0)
        return self.url_source

class Source(models.Model):
    name = models.CharField(max_length = 50)
    quality = models.IntegerField(default = 0)

    objects = SourceManager()

This works perfectly fine in the application - once source is created, the manager remembers it for it's existence as the sources do not change over their lifetime. However in tests they go away as the whole test is run in single transaction and then reverted.

What I find strange is that models.ForeignKey did not complain about getting non existing object, but the error appeared later, while sorting by source__quality as the underlaying JOIN SELECT could not find matching Source object.

Answer 2

I was not able to reproduce with sqlite3. Could it be that the instance of Reference passed in is not saved? The following ran without a hiccup:

def auto_create(instance):
    issue = Issue.objects.create(master = instance)
    print issue.references.count(), issue.references.all()
    assert issue.references.count()==0, "initial ref count is not null"
    assert len(issue.references.all())==0, "initial ref array is not empty"
    issue.references.add(instance)
    print issue.references.count(), issue.references.all()
    assert issue.references.count()==1, "ref count is not incremented"
    assert len(issue.references.all())==1, "initial ref array is not populated"


def test_auto():
    s = Source()
    s.save()
    r = Reference(source=s)
    r.save()
    auto_create(r)