How do I test a crawler? ie: mock web requests

Question

So I'm testing the individual parts of my crawler and now I've hit a stump: mocking the actual requests.

I don't want to make a request every time I run a spec.

Anyone ever tried starting a webrick server in the test helper?

Something like

myserver = SomeServerLib.start('localhost', some_port)
myserver.serve_directory(a_directory_with_some_html_files)

Thanks!

Answer 1

You can mock the HTTP request and use expectations. See Webmock project.

Answer 2

You can also use FakeWeb, which lets you provide canned HTTP responses during tests.

http://fakeweb.rubyforge.org/

(As a side note, you probably should look into alternatives if you're using Net::HTTP. It's very slow. Check out em-http-request )