cURL - Load a site with CloudFlare protection

Question

I have a site which I want to load using cURL. However, the site I'm talking about is using cloudflare protection. While on cURL request => it says 'checking your browser, you will be redirected in 5 seconds' and I'm getting redirected to the page that doesn't exists because as I think I didn't pass some protection test.

I've been looking for my answer for a long and i found only two useful links:

Link 1

Link 2

But still I can't figure it out. Any help for me?

Answer 1

That would be CloudFlare's I'm Under Attack Mode --> http://blog.cloudflare.com/introducing-im-under-attack-mode

If you are the owner of the site you should whitelist your own IP address. If you're a visitor to the site either

1.) The website owner has the feature enabled on their own website in which case it's intentional and there is no way around that as they are protecting their website or

2.) You are loading the site too often -- and you're triggering CloudFlare's DDOS protection.

p.s. Disclaimer: I work for CloudFlare.

Answer 2

You can get around the in-under-attack-mode by selecting 'Copy as cUrl' in the network section under developer tools of any major browser.

This copies all the required cookies so your curl can be 'authenticated'. How long these cookies last i would assume would be up to cloudflare.

SideNote: The curl must be run on the same IP as you were loading the site with. Also for any real test/developing environment I would suggest selenium and/or phantomjs which are alot better and easily scriptable languages for website testing.

curl 'https://ilikeurls.net/ourpage.php?do=command' \
    -H 'Host: ilikeurls.net' \
    -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:33.0) Gecko/20100101 Firefox/33.0' \
    -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
    -H 'Accept-Language: en-US,en;q=0.5' \
    -H 'Referer: https://ilikeurls.net/outpage.php' \                                                    
    -H 'Cookie: all required cookies will appear here' \
    -H 'Connection: keep-alive'\
    --compressed

Though you might be better off using selenium or phantomjs to script out these tests.

Answer 3

You might try adjusting your bot settings or finding a way to whitelist CURL.

All my curl requests started returning 403 errors after I adjusted Cloudflare settings at Firewall > Bot. All were permitted; I set Automated to Block and Likely Automated to Challenge and it broke CURL.