Avoiding cartesian product with Django

Question

I have the following three tables defined.

class Operator(models.Model):
    DisplayName = models.CharField(max_length=64)

    class Meta:
        app_label = "Experiment"
        db_table = "EXPERIMENT_OPERATOR"

class OperatorSummary(models.Model):
    Operator = models.ForeignKey(Operator, related_name="TransactionSummary")
    TransactionCount = models.IntegerField()
    TransactionValue = models.DecimalField(max_digits=18, decimal_places=2)
    StartTime = models.DateTimeField(default=timezone.now())

    class Meta:
        app_label = "Experiment"
        db_table = "EXPERIMENT_OPERATORSUMMARY"

class OperatorAlerts(models.Model):
    Operator = models.ForeignKey(Operator, related_name="AlertSummary")
    AlertScore = models.IntegerField()
    AlertCount = models.IntegerField()
    StartTime = models.DateTimeField(default=timezone.now())

    class Meta:
        app_label = "Experiment"
        db_table = "EXPERIMENT_OPERATORALERTS"

For an Operator, I would like to retrieve the AlertScore and the TransactionCount for a given date range. The query I'm using looks like this:

tz = timezone.get_default_timezone()    
vs = Operator.objects.filter(DisplayName="Jimmy",
                             TransactionSummary__StartTime__gte=tz.localize(datetime(year=2013, month=10, day=1)),
                             AlertSummary__StartTime__gte=tz.localize(datetime(year=2013, month=10, day=1)))\
    .annotate(TotalTransactions=Sum("TransactionSummary__TransactionCount"),
              TotalAlerts=Sum("AlertSummary__AlertScore"))\
    .values("DisplayName", "TransactionSummary__TransactionCount", "AlertSummary__AlertScore")

This query performs a cartesian product and returns all the rows in the OperatorAlerts and OperatorSummary table that match the query. This is what it returns:

{'AlertSummary__AlertScore': 20, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 10}
{'AlertSummary__AlertScore': 44, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 10}
{'AlertSummary__AlertScore': 543, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 10}
{'AlertSummary__AlertScore': 20, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 22}
{'AlertSummary__AlertScore': 44, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 22}
{'AlertSummary__AlertScore': 543, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 22}
{'AlertSummary__AlertScore': 20, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 234}
{'AlertSummary__AlertScore': 44, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 234}
{'AlertSummary__AlertScore': 543, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 234}

I'd like to fix this so that I get the following result:

{'AlertSummary__AlertScore': 607, 'DisplayName': u'Jimmy', 'TransactionSummary__TransactionCount': 266}

All the results are collapsed into one row where the AlertScore and TransactionCount are summed.

Is that possible? I can always fall back to making a separate query for the OperatorAlerts and OperatorSummary and then iterating over the result set in Python to get the results I want or calling .aggregate, but I'm sure there must be a better way?

Answer 1

Try reversing the order in which you apply the values() and annotate() methods. values() should come first:

vs = Operator.objects.filter(DisplayName="Jimmy",
                             TransactionSummary__StartTime__gte=tz.localize(datetime(year=2013, month=10, day=1)),
                             AlertSummary__StartTime__gte=tz.localize(datetime(year=2013, month=10, day=1)))\
    .values("DisplayName")\
    .annotate(TotalTransactions=Sum("TransactionSummary__TransactionCount"),
              TotalAlerts=Sum("AlertSummary__AlertScore"))

This will group the results by fields mentioned in values() and then generate an annotation for each group. The order is very significant - as documented.

Applying values() and annotate() in the way you do (i.e. annotate() before values()) will generate annotations for every item separately.

Please note that the code above will group the results by DisplayName. You may want to group by a different field, for example pk.

Also, I assume that in your real code you will want to get the values for multiple operators at once. If you always queried for one operator at a time (like you do in your example) you would be better of using aggregate() instead of annotate().