I had the same issue. To fix this we need a customized PageFetcher. You can find the sample here. http://code.google.com/p/crawler4j/issues/detail?id=174
crawl https pages with crawler4j
Question
For months now we used crawler4j to crawl a https site. Suddenly, since last friday, we're not able to crawl the very same https site. Has something changed in the https-protocol? The site is https://enot.publicprocurement.be/enot-war/home.do
As a test, just try to grab the title: Welkom op het platform e-Notification
Any help is much appreciated.
La solution 2
Autres conseils
I found it works best when setting CrawlConfig
CrawlConfig config = new CrawlConfig();
config.setIncludeHttpsPages(true);
config.setUserAgentString("Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36");
PageFetcher pageFetcher = new PageFetcher(config);
You could use this PageFetcher subclass in place of your PageFetcher. This solved all the issues for me.
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import javax.net.ssl.SSLContext;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import edu.uci.ics.crawler4j.crawler.CrawlConfig;
import edu.uci.ics.crawler4j.fetcher.PageFetcher;
public class PageFetcher2 extends PageFetcher {
public static final String DEFAULT_USER_AGENT = "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:45.0) Gecko/20100101 Firefox/45.0";
public static final RequestConfig DEFAULT_REQUEST_CONFIG = RequestConfig.custom().setConnectTimeout(30 * 1000)
.setSocketTimeout(60 * 1000).build();
public PageFetcher2(CrawlConfig config) throws KeyManagementException, NoSuchAlgorithmException, KeyStoreException {
super(config);
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(30);
connectionManager.setDefaultMaxPerRoute(30);
SSLContext sslContext = new SSLContextBuilder()
.loadTrustMaterial(null, (certificate, authType) -> true).build();
httpClient = HttpClients.custom()
.setSSLContext(sslContext)
.setSSLHostnameVerifier(new NoopHostnameVerifier())
.setConnectionManager(connectionManager)
.setUserAgent(DEFAULT_USER_AGENT)
.setDefaultRequestConfig(DEFAULT_REQUEST_CONFIG)
.build();
}
}
Licencié sous: CC-BY-SA avec attribution
Non affilié à StackOverflow