Scrapy: Use Feed Exporter to store Items without using return

Question

I'd like to use Scrapy to retrieve some data from a website with a table, and multiple pages. Here's what that looks like:

class ItsyBitsy(Spider):
    name = "itsybitsy"
    allowed_domains = ["mywebsite.com"]
    start_urls = ["http://mywebsite.com/Default.aspx"]

    def parse(self, response):
        # Performs authentication to get past the login form
        return [FormRequest.from_response(response,
            formdata={'tb_Username':'admin','tb_Password':'password'},
            callback=self.after_login,
            clickdata={'id':'b_Login'})]

    def after_login(self, response):
        # Session authenticated. Request the Subscriber List page
        yield Request("http://mywebsite.com/List.aspx", 
                  callback=self.listpage)

    def listpage(self, response):
        # Parses the entries on the page, and stores them
        sel = Selector(response)
        entries = sel.xpath("//table[@id='gv_Subsribers']").css("tr")
        items = []
        for entry in entries:
            item = Contact()
            item['name'] = entry.xpath('/td[0]/text()')
            items.append(item)

        # I want to request the next page, but store these results FIRST
        self.getNext10()
        return items

I'm stuck at that last line at the end. I want to request the next page (so that I can pull another 10 rows of data), but I want to save the data FIRST using the feed exporter (that's configured in my settings.py).

How can I tell the feed exporter to save the data without calling return items (which would prevent me from continuing to scrape the next 10 rows).

Answer 1

Answer: use generators.

def listpage(self, response):
    # Parses the entries on the page, and stores them
    sel = Selector(response)
    entries = sel.xpath("//table[@id='gv_Subsribers']").css("tr")
    items = []
    for entry in entries:
        item = Contact()
        item['name'] = entry.xpath('/td[0]/text()')
        yield item

    # remember get next has to return Request with callback=self.listpage
    yield self.getNext10()