Est-ce que les sites de limite de taux ScraperWiki il râpe?

Question

Est-ce que ScraperWiki en quelque sorte scraping limite de vitesse automatique, ou devrais-je ajouter quelque chose comme le sommeil (1 * random.random ()) à la boucle?

Answer 1

There is no automatic rate limiting. You can add a sleep command written in your language to add rate limiting.

Very few servers check for rate limiting, and usually servers containing public data don't.

It is, however, good practice to make sure you don't overrun the remote server. By default, scrapers only run in one thread, so there is a built in limit to the load you can produce.