ForeignKey to Django-MPTT tree generating storm of SQL requests in admin inline

Question

I'm faced with a dramatic performance issue in connection with django-mptt. Here is my case:

• I have a Quizz class
• I have a Question class with a FK to Quizz and a FK to a Category class
• I have a Category class that is an MPTT tree (because my categorizing is hierarchical)

Now, I have an actual Quizz with 7 Questions and an admin view that shows the Questions as inlines to the QuizzAdmin view, and the inlines bear the Category as Select field.

Then comes the trouble:

• I've had the questions loaded as prefetch_related (and even tried to have the questions__category) loaded like that
• in spite of this, I see my debug toolbar showing a series of 16 queries happening at template rendering time (template/edit_inline/tabular.html). On my dev laptop, that means 1 min to load this all (and on my test environment with actual data, that means 10 minutes !)

These 16 queries are a succession of the below: (pls note I'm in test with dummy categories)

SELECT "quizz_category"."id", "quizz_category"."parent_id", "quizz_category"."name", 
"quizz_category"."name_en", "quizz_category"."name_fr", "quizz_category"."lft",
"quizz_category"."rght", "quizz_category"."tree_id", "quizz_category"."level",
"quizz_category"."description", "quizz_category"."description_en",
"quizz_category"."description_fr" FROM "quizz_category" ORDER BY
"quizz_category"."tree_id" ASC, "quizz_category"."lft" ASC

and

SELECT "quizz_category"."id", "quizz_category"."parent_id", "quizz_category"."name", 
"quizz_category"."name_en", "quizz_category"."name_fr", "quizz_category"."lft",
"quizz_category"."rght", "quizz_category"."tree_id", "quizz_category"."level",
"quizz_category"."description", "quizz_category"."description_en",
"quizz_category"."description_fr" FROM "quizz_category" WHERE ("quizz_category"."lft" <= 3
AND "quizz_category"."rght" >= 6 AND "quizz_category"."tree_id" = 1 ) ORDER BY
"quizz_category"."lft" ASC

Any idea of what I could do to reduce the number of queries?

Thanks ahead LA

[EDIT 1]

There was a stupid thing that explains half of the issue: my Category's __unicode__() was looking at the object's parents' __unicode__() (fortunately my tree is only 2-level deep)

Now in my optimal configuration, I still have 9 times "SELECT ... FROM quizz_category" (no WHERE clause) for 8 entries, supposedly for building the choices of the Select field.

Anyone has an idea about how to get this query cached and only run once?

Note: my current optimal configuration is to have .select_related('category') in QuestionInline

---

class QuestionInline(admin.TabularInline): # admin.StackedInline
    model = Question
    extra = 0
    ordering = ['position',]

    def queryset(self, request):
        return super(QuestionInline, self).queryset(request).select_related('category')


class QuizzAdmin(admin.ModelAdmin):
    list_display = ["name","rating_scale"]
    inlines = [QuestionInline]
    fieldsets = (
        (None, {'fields': (('name'), ('type',), 'description',
                           'rating_scale' )}),
    )

    def queryset(self, request):
        if getattr(self,'is_change_list', False):
            # it's a changelist view, we don't need details on ForeignKey-accessible objects
            return super(QuizzAdmin, self).queryset(request)
        else:
            return super(QuizzAdmin, self).queryset(request).select_related('rating_scale')

    def changelist_view(self, request, extra_context=None):
        self.is_change_list = True
        return super(QuizzAdmin, self).changelist_view(request, extra_context)

class Category(AbstractAnalyticTreeCategory):
    description         = BusinessTextField(_("description"))  # basically a text field of mine

    tree = AnalyticTreeManager()

    def __unicode__(self):
        return self.name

class Quizz(models.Model):
    name                = models.CharField(_("name of the quizz"), unique=True, max_length=60)
    description         = BusinessTextField(_("description"))
    type                = models.CharField(_("type"), choices=QUIZZ_TYPE_CHOICES, default=QUIZZ_SELF_EVALUATION, null=False, blank=False, max_length=2)
    rating_scale        = models.ForeignKey(MCQScale, verbose_name=_("applicable rating scale"), on_delete=models.PROTECT)


    def __unicode__(self):
        return self.name



class Question(models.Model):
    position = models.IntegerField(verbose_name=_("order index"), help_text=_("Order in which the question will appear."))
    quizz               = models.ForeignKey(Quizz, verbose_name=_("Related quizz"), null=False, blank=False, related_name='questions')
    title               = BusinessCharField(_("item"), max_length=60, null=True, blank=True)
    text                = BusinessTextField(_("question text"),)
    category            = TreeForeignKey(Category, verbose_name=_("dimension"), null=True, blank=False, on_delete=models.SET_NULL)

    def __unicode__(self):
        return self.title

Here is what the debug toolbar says about these queries (all the same):

SELECT "quizz_category"."id", "quizz_category"."parent_id", "quizz_category"."name", "quizz_category"."name_en", "quizz_category"."name_fr", "quizz_category"."lft", "quizz_category"."rght", "quizz_category"."tree_id", "quizz_category"."level", "quizz_category"."description", "quizz_category"."description_en", "quizz_category"."description_fr" FROM "quizz_category" ORDER BY "quizz_category"."tree_id" ASC, "quizz_category"."lft" ASC 3,68816058264% 1,66 Sel Expl Connection: default Isolation Level: Read committed Transaction Status: In transaction /Library/Python/2.7/site-packages/django/contrib/staticfiles/handlers.py in call(72) return self.application(environ, start_response) /Library/Python/2.7/site-packages/django/contrib/admin/widgets.py in render(263) output = [self.widget.render(name, value, *args, **kwargs)] 49

{{ field.contents|linebreaksbr }}

50 {% else %} 51
{{ field.field.errors.as_ul }} 52
{{ field.field }} 53
{% endif %} 54
55
{% endfor %} /Library/Python/2.7/site-packages/django/contrib/admin/templates/admin/edit_inline/tabular.html

Answer 1

So.. I found a solution, which is inspired from Caching queryset choices for ModelChoiceField or ModelMultipleChoiceField in a Django form and which I describe in that post.

Django admin has a strange overhead of 1 query due to the inline factory mechanism (I haven't gone in depth into this). This explains why in the normal case you have 2*k + 1 queries (k=number of items in the inline formset).

issue solved hopefully.

LAI

Answer 2

I'm using django 1.10 and none of the solutions in Caching queryset choices for ModelChoiceField or ModelMultipleChoiceField in a Django form worked for me. I ended using a different solution based on django-mptt's tree_item_iterator. My form went from 560 SQL queries to 10. My code:

from categories.models import Category
from mptt.utils import tree_item_iterator


def get_tree_choices(queryset, level_indicator='+----', ancestors=False):
    choices = []
    for node, tree in tree_item_iterator(queryset, ancestors=ancestors):
        name=''
        if ancestors:
            for i in tree['ancestors']:
                name+=level_indicator
            name+=' %s' % node.name
        else:
            name = str(node)
        choices.append((node.id, name))
    return choices


# Register your models here.
class CompanyAdminForm(forms.ModelForm):
    category=forms.ChoiceField(choices=get_tree_choices(Category.objects.all()))

    error_messages = {
        'invalid_vat_id': _('Invalid Portuguese VAT ID')
    }

    class Meta:
        exclude = ('category',)
        ...

    def save(self, commit=True):
        company=super(CompanyAdminForm, self).save(commit=False)
        company.category=Category.objects.get(id=self.cleaned_data['category'])
        company.save()
        return company
    ...


@admin.register(Company)
class CompanyAdmin(admin.ModelAdmin):
        ...

Still not a 100% satisfied but at least it's it's quick.