Google image search by URL, unable to crawl the page

Question

I am trying to use Google image search by url for research purposes. That was after I gave up searching by the actual image since I can't really get it to work.

What is done so far

Using PHP, I can issue an HTTP request to https://www.google.com/searchbyimage?image_url=https://www.google.com/images/nav_logo117.png

If you copy that URL into your browser, you see the results (seems some automatic redirects take place here).

However, if you try to manually request the URL, through PHP for example, or through http://web-sniffer.net/?url=images.google.com/searchbyimage?image_url=https://www.google.com/images/nav_logo117.png you get an HTTP 302 error that says that the page has been moved to some other URL.

I extracted that URL, tried it out on the browser and again it works. But again if you try this URL manually you get another HTTP 302 error, which finally takes you to Google's homepage.

I've seen questions like: Script to use Google Image Search with local image as input which seem to have been able to do it, but the asker didn't run into the same problem I did

Answer 1

It seems that Google is performing a redirection based on the User-Agent in your request. So if you include a real User-Agent - something copied from a real web browser - the request should work correctly.

Here is some example php code that works for me:

$location = 'http://www.google.com/searchbyimage?image_url=https://www.google.com/images/nav_logo117.png';

do {
  $ch = curl_init($location);

  curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.0; rv:20.0) Gecko/20100101 Firefox/20.0');
  curl_setopt($ch, CURLOPT_HEADER, true);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

  $data = curl_exec($ch);
  $http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
  curl_close($ch);

  $data = str_replace("\r\n","\n",$data);
  list($headers, $data) = explode("\n\n", $data, 2);
  $headers = explode("\n",$headers);

  $location = null;
  foreach ($headers as $header)
    if (stripos($header, 'Location:', 0) === 0)
      $location = trim(substr($header,9));

} while ($http_code == 302 && $location != null);

echo $data;