Django Database Optimization? Is this slow or normal?

Question

I have a few questions related with database performance. I have the following Django model and its corresponding admin class in my app. The database is MySQL and it is hosted at Amazon RDS.

1. It took more than 20 minutes to add around 45000 records in the Notifications table, via a for loop. Is this time slow or normal?
2. The django admin interface for this table is too slow. It takes around 30 seconds to load under no dB load. And it usually takes more than 2 minutes to load when the database is performing any job. This table usually adds one million records every week or two. Is there a way to improve the performance of the database and/or system, or is this current load time normal?

MODEL

class Notification(models.Model):
    id = models.AutoField(primary_key=True)
    token = models.ForeignKey(Token, blank=False, null=False)
    alert = models.ForeignKey(Alert, blank=False, null=False)
    created_at = models.DateTimeField(auto_now=True)
    is_sent = models.BooleanField(default=False)
    is_processed = models.BooleanField(default=False)
    error_sending = models.BooleanField(default=False)
    # ...
    def __unicode__(self):
        return u'%s' % (self.alert )

ADMIN

class AppNotification(admin.ModelAdmin):
    fields = ['token','alert','is_sent','is_processed','error_sending']

    #
    list_display = ('token','alert','created_at','is_sent','is_processed','error_sending')

    #
    search_fields = ('app__app_name','token__token')

    #
    list_select_related = True

    #
    list_per_page = 25

admin.site.register(Notification,AppNotification)

Answer 1

It took more than 20 minutes to add around 45000 records in the Notifications table, via a for loop. Is this time slow or normal?

It's not unusual in this setup. Django ORM is not fast and is not a good way to add thousands of entries if speed is important. Speed of RDS depends on the instance type, but generally lower-end ones aren't fast either.

Alternative is going low-level SQL or using bulk_create, however you have to keep in mind that neither of these methods will call objects .save() nor will it send pre_save and post_save signals. Which in case of these containing lot of business logic, that can be a problem.

The django admin interface for this table is too slow. It takes around 30 seconds to load under no dB load. And it usually takes more than 2 minutes to load when the database is performing any job. This table usually adds one million records every week or two. Is there a way to improve the performance of the database and/or system, or is this current load time normal?

You've got list_select_related = True in your model admin, which seems the only thing that can make it slow. It does join with Tokens and Alerts.

Answer 2

Regarding your second question:

The django admin interface for this table is too slow. It takes around 30 seconds to load under no dB load. And it usually takes more than 2 minutes to load when the database is performing any job.

That seems like a slow query? A good place to start would be to see which queries are being run (django debug toolbar) and use the tools your db provides explain to debug them and add appropriate indexes

---

in django 1.4 bulk_create method has been added which will at least cut down on time spent transferring data over the network for your inserts?

Answer 3

I was experiencing the exact same problems recently and had to dig deep to find work-arounds to get my Django admin humming along for tables with 100M+ records. My problem ended up being expensive COUNT(*) queries on page load and also poorly constructed searching queries whenever I used one of my search_fields from the ModelAdmin.

After a lot of research and pondering I came up with a few nice solutions and documented them here: http://craiglabenz.me/2013/06/12/how-i-made-django-admin-scale/