Web Scraping Automation programs [closed]

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

---

Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.

Closed 8 years ago.

Improve this question

This may be off topic but if anyone can help or point me in the right direction.....or the correct place to put this question?

Can anyone suggest a good Automated Web Scraping program for use on a windows machine? I would like to be able to automate the process and set it to be able to "scrape" the site every 5 mins or so and have the resulting xml exported to another website or database?

Answer 1

If the actual process is taking too long, those 10 seconds or more likely to being the network latency rather than actual parsing and scraping its self. If that is the case, you probably want to have them executing in parallel rather than sequentially if you have not already done so.

It would help if you had a more specific issue as there is several ways about going about this. You could have it still parsing on the phone. If that is too problematic for whatever reason, in my opinion, I would instead build a separate HTTP REST server for the phone client to have it interact with to get the data it requires.

There is many ways about going about this, and these are simply recommendations. If you want a more specific answer, you'll need a more specific question. Consider then asking a more speific programming issue if you run into one.