GridSearchCV vs RandomSearchCV and How it works?

Question

GridSearchCV vs RandomSearchCV

Can somebody explain in-detailed differences between GridSearchCV and RandomSearchCV? And how the algorithms work under the hood?

As per my understanding from the documentation:

RandomSearchCV

This uses a random set of hyperparameters. Useful when there are many hyperparameters, so the search space is large. It can be used if you have a prior belief on what the hyperparameters should be.

GridSearchCV

Creates a grid over the search space and evaluates the model for all of the possible hyperparameters in the space. Good in the sense that it is simple and exhaustive. On the minus side, it may be prohibitively expensive in computation time if the search space is large (e.g. very many hyper parameters).

Answer 1

Imagine the following scenario:

params = {
   epoch = [20, 30, 40, 50], #those numbers are only for example
   dense_layer_size = [20, 30], 
   second_danse_layer = [30, 40]    
}

In GridSearch you try all the combinations of your parameters, in this case:

(4 * 2 * 2) = 16 #Total of parameters

In RandomSearch as in documentation:

Not all parameter values are tried out, but rather a fixed number of parameter settings is sampled from the specified distributions. The number of parameter settings that are tried is given by n_iter.

In this case, he select "n_iter" number of combinations and try then. Is good for optimizing less parameters, but, in cases when you don't have sure about multiples parameters, may let aside some combinations that would be better.

Another good approach, is Using an genetic algorithm to optimize your network.

Genetic algorithm generate a few combinations of "individuals" (combination of parameters) and at each step, select the best individuals(called parents) and crossover then, generating more individuals with the characteristic from both parents. This way, in some case you can optimize your network and add random elements when you don't have sure about what parameter would be increase your result.

If you want an easy to integrate genetic algorithm you can check this project.

Answer 2

Let's start with GridSearch: GridSearchCV taks a dictionary of parameters like:

param = {'gamma': [0.1,0.001,0.0001], 'C': [1,10,100,1000]}

and runs n models where n is the count of all parameter combinations.

All combinations combined is often refered to as the parameter space. Using GridSearchCV can take a lot computational wise since it has to train your model for each combination, often including cross validation. So for each combination it would train on k folds. This can often explode your computation time tremendously. Benefit however is that if you run it on a broad parameter space you will get the "best" parameter settings possible.

RandomSearchCV now takes your parameter space and picks randomly a predefined number of times and runs the model that many times. You can even give him continuous distributions for parameters to randomly pick values from. That way you have a computation optimized way of experimenting on random parameter settings. This is usefull if you already have a general idea on which hyperparameters you want to tune but are not yet quite sure which values to use.

Answer 3

There's rather exaustive explanation in scikit's docs. It's a good idea to run quite wide grid search first, to get some intuition of feasible hyper-params space. For fine tuning do random search later with narrowed down scope.