Question

Like we go on google and perform google search via keywords.

Can we do such query programaticaly? like

http://www.google.com/search?q=cupertino+american+food

After executing the query we should get all search result details for each link to store in database.

Exactly like some site provide REST api access, so that user can get bunch of results his query.

I don't have seen something like this possible with google or not.

Was it helpful?

Solution 2

@mahtOrz: Okay, here's some rough code that will deliver a Json back to the console. Note that the api base search string is different than the one you have which is www.google.com/search?q=cupertino+american+food. You need to use the Google API base URL below. Do you have your APIkey and CxKey? If not, I can walk you through those steps too.

using System;
using System.Text;
using System.Net;
using System.IO;
using System.Web;

namespace GoogleSearchTest1
{
    class Program
    {
        //Google keys                                                   
        const string APIKey = "{your key here}";
        const string CSEKey = "{your key here}";
        //base url for the search query                                                 
        const string GoogleBaseURL = "https://www.googleapis.com/customsearch/v1?"; //per Google documentation

    public static void Main(string[] args)
    {
        string myQuery = "cupertino american food"; //put what you're searching for here
        string result = submitSearch(myQuery);
        Console.WriteLine(result);
        string dummy = Console.ReadLine();
    }
    public static string submitSearch(string myQuery)
    {
        try
        {    
            string final = string.Format(GoogleBaseURL+"key={0}&cx={1}&q={2}",
                HttpUtility.UrlEncode(APIKey),
                HttpUtility.UrlEncode(CSEKey),
                HttpUtility.UrlEncode(myQuery));
            final += "&alt=json";
            WebRequest myRequest = WebRequest.Create(final);
            HttpWebResponse myResponse = (HttpWebResponse)myRequest.GetResponse();
            Stream myStream = myResponse.GetResponseStream();
            StreamReader myReader = new StreamReader(myResponse.GetResponseStream(), System.Text.Encoding.UTF8);
            string result = myReader.ReadToEnd();
            myStream.Close();
            myReader.Close();
            myResponse.Close();
            return result;
        }
            catch (Exception e)
        {
            //debug statement       
        }           
        return null;
    }
}

}

OTHER TIPS

Whatever technique you're going to use, Google will block your IP for bot-like search queries. And don't try with TOR proxy because all their IPs are always banned or challenged with captcha.

You have to use Google API in order to be compliant with Google's T&C. Also the result is much much better

https://developers.google.com/custom-search/json-api/v1/overview

The API is free if you have a CSE and has a limit of 100 queries per day. If you need more you'll be billed 5$ per 1000 queries

use cUrl request, hand in hand with output buffering

@user123: I can offer some tips if you can work in C#? The API steps are extensive. Let me know!

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top