crawler4j always returns fatal transport error

Question

This is what I get for any seed I add to crawler4j.

ERROR [Crawler 1] Fatal transport error: Connection to http://example.com refused while fetching http://example.com/page.html (link found in doc #0)

This is really weird for me. I don't know what causes it.

Answer 1

try to increase the politeness setting of the crawler. It always happens to me when the server that I want to crawler is not good.