What is the best way to crawl a login based sites?

Question

I've to automate a file download activity from a website (similar to, let's say, yahoomail.com). To reach a page which has this file download link, i've to login, jump from page to page to provide some parameters like dates etc., and finally click on download link.

I am thinking of three approaches:

1. Using WatIN and develop a windows service that periodically executes some WatiN code to traverse through the page and download the file.

2. Using AutoIT (no much idea)

3. Using a simple HTML parsing technique (there are several questions here eg., how to maintain a session after doing a login? how to do a logout after doing it?

Answer 1

Try a Selenium script, automated with Selenium Remote Control.

Answer 2

I use scrapy.org, it's a python library. It's quiet good actually. Easy to write spiders and it's very extensive in it's functionality. Scraping sites after login is available in the package.

Here is an example of a spider that would crawl a site after authentication.

class LoginSpider(BaseSpider):
    domain_name = 'example.com'
    start_urls = ['http://www.example.com/users/login.php']

    def parse(self, response):
        return [FormRequest.from_response(response,
                formdata={'username': 'john', 'password': 'secret'},
                callback=self.after_login)]

    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.log("Login failed", level=log.ERROR)
            return

        # continue scraping with authenticated session...

Answer 3

I used mechanize for Python with success for a few things. It's easy to use and supports HTTP authentication, form handling, cookies, automatic HTTP redirection (30X), ... Basically the only thing missing is JavaScript, but if you need to rely on JS you're pretty much screwed anyway.

Answer 4

Free Download Manager is great for crawling, and you could use wget.