Early-Stopping for logistic regression. Theano

Question

I am trying to understand the code for the logistic regression on the official documentation, but I am struggling to understand the logic behind this code:

# early-stopping parameters
patience = 5000  # look as this many examples regardless
patience_increase = 2     # wait this much longer when a new best is
                              # found
improvement_threshold = 0.995  # a relative improvement of this much is
                               # considered significant
validation_frequency = min(n_train_batches, patience/2)
                              # go through this many
                              # minibatches before checking the network
                              # on the validation set; in this case we
                              # check every epoch

best_params = None
best_validation_loss = numpy.inf
test_score = 0.
start_time = time.clock()

done_looping = False
epoch = 0
while (epoch < n_epochs) and (not done_looping):
    # Report "1" for first epoch, "n_epochs" for last epoch
    epoch = epoch + 1
    for minibatch_index in xrange(n_train_batches):

        d_loss_wrt_params = ... # compute gradient
        params -= learning_rate * d_loss_wrt_params # gradient descent

        # iteration number. We want it to start at 0.
        iter = (epoch - 1) * n_train_batches + minibatch_index
        # note that if we do `iter % validation_frequency` it will be
        # true for iter = 0 which we do not want. We want it true for
        # iter = validation_frequency - 1.
        if (iter + 1) % validation_frequency == 0:

            this_validation_loss = ... # compute zero-one loss on validation set

            if this_validation_loss < best_validation_loss:

                # improve patience if loss improvement is good enough
                if this_validation_loss < best_validation_loss * improvement_threshold:

                    patience = max(patience, iter * patience_increase)
                best_params = copy.deepcopy(params)
                best_validation_loss = this_validation_loss

        if patience <= iter:
            done_looping = True
            break

Could any one, explain to me, what do the variables: patience, patience_increase, improvement_threshold, validation_frequency, iter, represent ?

What this condition do ?

if (iter + 1) % validation_frequency == 0:

Answer 1

Patience is the number of training batches to do before you stop. iter is the number of training batches you've seen. Each iteration, you decide whether or not your validation is lower than your previous best. The improvement is only stored if the score is lower than improvement_threshold * validation_score.

It appears that patience_increase is a multiplier. Every time you have a new lowest score, you up the total number or training batches to iter*patience_increase, but not below the current value of patience.

validation_frequency is just the number of batches between times you check the validation score.