Django - Displaying result information while optimizing database queries with models that multiple foreign key relationships

Question

So I'm trying to put together a webpage and I am currently have trouble putting together a results page for each user in the web application I am putting together.

Here are what my models look like:

class Fault(models.Model):
    name = models.CharField(max_length=255)
    severity = models.PositiveSmallIntegerField(default=0)
    description = models.CharField(max_length=1024, null=False, blank=False)
    recommendation = models.CharField(max_length=1024, null=False, blank=False)
    date_added = models.DateTimeField(_('date added'), default=timezone.now)
    ...


class FaultInstance(models.Model):
    auto = models.ForeignKey(Auto)
    fault = models.ForeignKey(Fault)
    date_added = models.DateTimeField(_('date added'), default=timezone.now)

    objects = FaultInstanceManager()
    ...

class Auto(models.Model):
    label = models.CharField(max_length=255)
    model = models.CharField(max_length=255)
    make = models.CharField(max_length=255)
    year = models.IntegerField(max_length=4)
    user = models.ForeignKey(AUTH_USER_MODEL)
    ...

I don't know if my model relationships are ideal, however it made sense it my head. So each user can have multiple Auto objects associated to them. And each Auto can have multiple FaultInstance objects associated to it.

In the results page, I want to list out the all the FaultInstances that a user has across their Autos. And under each listed FaultInstance I will have a list of all the autos that the user owns that has the fault, with its information (here is kind of what I had in mind).

All FaultInstance Listing Ordered by Severity (large number to low number)

FaultInstance:
   FaultDescription:
   FaultRecommendation:
   ListofAutosWithFault:
       AutoLabel AutoModel AutoYear ...
       AutoLabel AutoModel AutoYear ...

Obviously, do things the correct way would mean that I want to do as much of the list creation in the Python/Django side of things and avoid doing any logic or processing in the template. I am able to create a list per severity with the a model manager as seen here:

class FaultInstanceManager(models.Manager):
    def get_faults_by_user_severity(self, user, severity):
        faults = defaultdict(list)
        qs_faultinst = self.model.objects.select_related().filter(
                auto__user=user, fault__severity=severity
            ).order_by('auto__make')
        for result in qs_faultinst:
            faults[result.fault].append(result)

        faults.default_factory = None
        return faults

I still need to specify each severity but I guess if I only have 5 severity levels, I can create a list for each severity level and pass each individual one to template. Any suggestions for this is appreciated. However, thats not my problem. My stopping point right now is that I want to create a summary table at the top of their report which can give the user breakdown of fault instances per make|model|year. I can't think of the proper query or data structure to pass on to the template.

Summary (table of all the FaultInstances with the following column headers):

FaultInstance    Make|Model|Year    NumberOfAutosAffected

This will let me know metrics for a make or a model or a year (in the example below, its separating faults based on model). I'm listing FaultInstances because I'm only listed Faults that a connected to a user.

For Example

Bad Starter     Nissan     1
Bad Tailight    Honda      2
Bad Tailight    Nissan     1

And I am such a perfectionist that I want to do this while optimizing database queries. If I can create a data structure in my original query that will be easily parsed in template and still get both these sections in my report (maybe a defaultdict of a defaultdict(list)), thats what I want to do. Thanks for the help and hopefully my question is thorough and makes sense.

Answer 1

It makes sense to use related names because it simplifies your query. Like this:

class FaultInstance(models.Model):
    auto = models.ForeignKey(Auto, related_name='fault_instances')
    fault = models.ForeignKey(Fault, related_name='fault_instances')
    ...

class Auto(models.Model):
    user = models.ForeignKey(AUTH_USER_MODEL, related_name='autos')

In this case you can use:

qs_faultinst = user.fault_instances.filter(fault__severity=severity).order_by('auto__make')

instead of:

qs_faultinst = self.model.objects.select_related().filter(
                auto__user=user, fault__severity=severity
            ).order_by('auto__make')

I can't figure out your summary table, may be you meant:

Fault    Make|Model|Year    NumberOfAutosAffected

In this case you can use aggregation. But It (grouping) would still be slow if you have enough data. The one easy solution is just to denormalize data by creating extra model and create few signals to update it or you can use cache.

If you have a predefined set of severities then think about this:

class Fault(models.Model):

    SEVERITY_LOW = 0
    SEVERITY_MIDDLE = 1
    SEVERITY_HIGH = 2
    ...
    SEVERITY_CHOICES = (
        (SEVERITY_LOW, 'Low'),
        (SEVERITY_MIDDLE, 'Middle'),
        (SEVERITY_HIGH, 'High'),
        ...
    )
    ...
    severity = models.PositiveSmallIntegerField(default=SEVERITY_LOW,
                                                choices=SEVERITY_CHOICES)
    ...

In your templates you can just iterate through Fault.SEVERITY_CHOICES.

Update:

Change your models:

Аllocate model into a separate model:

class AutoModel(models.Model):
    name = models.CharField(max_length=255)

Change the field model of model Auto :

class Auto(models.Model):
    ...
    auto_model = models.ForeignKey(AutoModel, related_name='cars')
    ...

Add a model:

class MyDenormalizedModelForReport(models.Model):

    fault = models.ForeignKey(Fault, related_name='reports')
    auto_model = models.ForeignKey(AutoModel, related_name='reports')
    year = models.IntegerField(max_length=4)
    number_of_auto_affected = models.IntegerField(default=0)

Add a signal:

def update_denormalized_model(sender, instance, created, **kwargs):
    if created:
        rep, dummy_created = MyDenormalizedModelForReport.objects.get_or_create(fault=instance.fault, auto_model=instance.auto.auto_model, year=instance.auto.year)
        rep.number_of_auto_affected += 1
        rep.save()

post_save.connect(update_denormalized_model, sender=FaultInstance)