optimizing django architecture to speed up list view within admin

Question

I am developing a medium site for my company, with large amounts of data (research publications, hundreds of employees, etc.) and security constraints which made me think of using django-guardian to handle object permissions. But now I'm realising that this may be slow within the admin.

We have already implemented a redis cache which seems to work fairly ok, but still loading large list views (hundreds of elements) take loong time.

We are using, so far, the following set up:

django             1.5.5
django-cms         2.4.3
django-redis-cache 0.10.2
django-guardian    1.1.1
hiredis            0.1.2
redis              2.9.1

python             2.7.5
postgresql
centos

As an example, this is the Person module, which list view takes ages to load (not as a superuser, in that case it's quite fast: this is the reason why I think the problem lies on the django-guardian multiple relations):

class Person(models.Model):
    TYPE_CHOICES = (
        ('S', _('Student')),
        ('E', _('Researcher')),
    )

    class Meta:
        permissions = (
            ('view_person', _('View person')),
        )
        index_together = (
            ('last_name', 'first_name'),
        )

    # Relations with other entities
    topics = models.ManyToManyField('topics.Topic', blank=True, related_name='people')
    competences = models.ManyToManyField('staff.Competence', blank=True, related_name='people', db_index=True)

    # Person properties
    cmsuser = models.OneToOneField(User, blank=True, related_name='person', null=True, db_index=True)
    sebra_username = models.CharField(max_length=20, blank=True, db_index=True)
    first_name = models.CharField(_('first name'), max_length=30, blank=True, db_index=True)
    last_name = models.CharField(_('last name'), max_length=30, blank=True, db_index=True)
    email = models.EmailField(_('email address'), blank=True, db_index=True)
    username = models.CharField(_('username'), max_length=30, unique=True, db_index=True)
    avatar = models.ImageField(upload_to='img/avatar/', blank=True)
    web = models.URLField(_("web site"), blank=True)
    cristin_profile = models.URLField(_('link to cristin profile'), blank=True)
    twitter = models.CharField(_("twitter username"), max_length=20, blank=True)
    telephone = models.CharField(
        blank=True,
        max_length=validators.MAX_LENGTH_PHONE,
        validators=[validators.validate_phone_format]
    )
    telephone_country_code = models.ForeignKey(Country, null=True, blank=True, related_name='phone_person')
    mobile = models.CharField(
        blank=True,
        max_length=validators.MAX_LENGTH_PHONE,
        validators=[validators.validate_phone_format]
    )
    mobile_country_code = models.ForeignKey(Country, null=True, blank=True, related_name='mobile_person')
    address = models.TextField(max_length=255, blank=True)
    cv = models.FileField(_('Curriculum Vitae'), upload_to='attachments/cv/', blank=True)
    vcard = models.FileField(_('Vcard'), upload_to='attachments/vcard/', blank=True)
    person_type = models.CharField(choices=TYPE_CHOICES, max_length=1, blank=True)
    extract = RichTextField(_('person extract'), blank=True, default='')
    slug = AutoSlugField(populate_from=('first_name', 'last_name'))

def __unicode__(self):
    if len(self.first_name) + len(self.last_name):
        return '%s %s' % (self.first_name, self.last_name)
    return self.username

def clean(self):
    super(Person, self).clean()
    if self.sebra_username.strip():
        # here goes validation and checks on the related objects

    def get_absolute_url(self):
        if PersonDepartmentMembership.objects.filter(active__exact=True, person__exact=self):
            return reverse('staff:profile_slug', kwargs={'slug': self.slug})
        return ''

I understand that the bottleneck might be as well within my Admin class. This is the one we're using:

class PersonAdmin(ModelAdmin):
    fields = (
        'sebra_username', 'first_name', 'last_name', 'avatar', 'email', 'person_type', 'extract',
        'topics', 'competences', 'web', 'cristin_profile', 'twitter', 'telephone_country_code',
        'telephone', 'address', 'mobile_country_code', 'mobile', 'cv', 'vcard'
    )
    search_fields = ('sebra_username', 'first_name', 'last_name', 'email', 'departments__name')
    list_filter = ('departments__name', 'research_groups__group_name', 'projects__project_name')

    inlines = (SomeInline,)

    class Media:
        js = (
            settings.STATIC_URL + 'js/jquery-1.9.0.min.js',
            settings.STATIC_URL + 'js/jquery-ui-1.9.2.custom.min.js',
            'modeltranslation/js/tabbed_translation_fields.js',
        )
        css = {
            'screen': ('modeltranslation/css/tabbed_translation_fields.css',),
        }

    def formfield_for_manytomany(self, db_field, request=None, **kwargs):
        if db_field.name == 'topics':
            kwargs['queryset'] = get_objects_for_user(user=request.user, perms=('topics.view_topic',))
        elif db_field.name == 'competences':
            kwargs['queryset'] = get_objects_for_user(user=request.user, perms=('staff.view_competence',))
        return super(PersonAdmin, self).formfield_for_manytomany(db_field, request, **kwargs)

    def queryset(self, request):
        if request.user.is_superuser:
            return super(PersonAdmin, self).queryset(request)
        return get_objects_for_user(user=request.user, perms=('staff.change_person',)).order_by('last_name')

    def has_add_permission(self, request):
        return request.user.has_perm('staff.add_person')

    def has_delete_permission(self, request, obj=None):
        return request.user.has_perm('staff.delete_person', obj)

    def has_change_permission(self, request, obj=None):
        return request.user.has_perm('staff.change_person', obj)

Could you give me any advice or suggest me any possible solution which we could integrate within the admin interface? :-)

Thanks in advance!

EDIT:

Using django-debug-toolbar, I can see that there are very few queries to django-guardian, and quite fast (all below 6ms). On the other hand, I have more than 7500 queries for a listview of 263, which slow the view up to 46 seconds to generate. Almost all of them are within my defined model, to load (I think) useless data: I suppose that just the name and object id are needed.

How can I limit the amount of data loaded within the queryset() method? Thanks.

Answer 1

If you're executing 7500 queries, your problem is probably that you're not loading related objects that you need - here's what I'd look at:

1. Do you have methods on your model that access related objects?
2. Do you have items in your list_display setting that access related objects?

Either eliminate calls to those things, or look at select_related.

Also - try changing the pagination of your admin views - if you reduce the number of records shown - how many fewer queries are there? That'll give you a clue as to how many of these problems you have.