Django: paginated ListView database performance

Question

I have a ListView with pagination:

class StoriesListView(ListView):
    model = Story
    paginate_by = 20

    def get_queryset(self):
        return Story.objects.all().order_by('-updated_at')

I have 1000 Story objects in the database. What happens when the user loads my view? Will all the 1000 be queried from the database or only 20? How can I optimize this?

Answer 1

it's depend how you are using it. QuerySet objects are lazy, and in this particular case the sql query will add LIMIT and OFFSET, so the query will always return only 20 results. But when you want using for example related objects in your template, you have to optimize your query with select_related or prefetch_related methods.

I think you should read how to optimize database access in django framework.

Hope this helps.

Answer 2

The Paginator class takes the return value of get_queryset (i.e. the entire queryset in this case) and splices it to give you access to only 20. It needs to operate on the entire queryset otherwise you wouldn't be able to use it to show a list of pages for example.

You can see follow the code to see it in action:

get() > get_context_data() > paginate_queryset() > Paginator.init()

This means that in your template context, the queryset variable is the entire queryset. The page variable is used to get only those objects that belong to the current page. This is done by splicing the initial queryset which will evaluate it and hit the DB. You can iterate over the objects in that page: {% for object in page %} and this won't hit the database again:

# https://github.com/django/django/blob/master/django/core/paginator.py#L119
def __getitem__(self, index):
    if not isinstance(index, (slice,) + six.integer_types):
        raise TypeError
    # The object_list is converted to a list so that if it was a QuerySet
    # it won't be a database hit per __getitem__.
    if not isinstance(self.object_list, list):
        self.object_list = list(self.object_list)
    return self.object_list[index]

Answer 3

It will only get 20 objects. Paginator takes result of get_queryset method as starting queryset, and it will only hit database when iterated so you're good. get_queryset method doesn't hit the db itself.

Answer 4

The same problem i have fetched during pagination. For pagination you don't need to make function inside modal.

you can use LIMT the same think can done by simple my generalize function

def testPagination(request):

dataObj             = paginationData('Item',page_no,10)
data["dataIndex"]   = dataObj["dataIndex"]
data["page_no"]     = dataObj["page_no"] #given page number will helps to active link at template
data["data"]        = dataObj["data"]  #actual data will helps in template
data['pagination']  = dataObj["pagination"] #Will helps to make dynamic link for pagination

GENERALIZE PAGINATION FUNCTION

def paginationData(modalName,pageNo,perPage):
    data = {} #initialize variables
    Modal = getModel(modalName) # Function to get modal reference
    count = Modal.objects.count()#Calculate no of records
    no_of_record_per_page = ceil(float(count) / perPage)
    print ",Records counts : " + str(no_of_record_per_page)
    pages = [] #No of pages
    pages.append(1) 
    a = 1
    while a < (no_of_record_per_page):
        a += 1 # Same as a = a + 1
        pages.append(a)
    data["pagination"] = pages # pagenation for display link of page
    data["page_no"] = pageNo
    data['data'] = Modal.objects.all()[((pageNo-1)*perPage):(pageNo*perPage)]
    data['dataIndex'] = str((pageNo -1)*perPage) 
    return data