I am working on a support vector machine, using sci-kit learn in Python.
I have trained the model, used GridSearch and cross-validation to find the optimal parameters, and have
evaluated the best model on a 15% holdout set.
The confusion matrix at the end says I have 0 misclassifications.
Later the model gave me incorrect predictions when I give it a handwritten digit (I haven't included the code for this, to keep this question respectfully short).
Because the SVM has zero error and further, later on it can't predict correctly, I have built this SVM incorrectly.
My question is this:
Am I right to suspect I used Cross Validation along with GridSearch somehow incorrectly? Or have I given GridSearch parameters that are somehow ridiculous, and are giving me false results?
Thanks for your time and effort for reading this far.
STEP 1: split the data set into 85%/15% using the train_test_split function
X_train, X_test, y_train, y_test =
cross_validation.train_test_split(X, y, test_size=0.15,
random_state=0)
STEP 2: apply the GridSearchCV function to the training set to tune the classifier
C_range = 10.0 ** np.arange(-2, 9)
gamma_range = 10.0 ** np.arange(-5, 4)
param_grid = dict(gamma=gamma_range, C=C_range)
cv = StratifiedKFold(y=y, n_folds=3)
grid = GridSearchCV(SVC(), param_grid=param_grid, cv=cv)
grid.fit(X, y)
print("The best classifier is: ", grid.best_estimator_)
The output is here:
('The best classifier is: ', SVC(C=10.0, cache_size=200,
class_weight=None, coef0=0.0, degree=3,
gamma=0.0001, kernel='rbf', max_iter=-1, probability=False,
random_state=None, shrinking=True, tol=0.001, verbose=False))
STEP 3: Finally, evaluate the tuned classifier on the remaining 15%
hold-out set.
clf = svm.SVC(C=10.0, cache_size=200, class_weight=None, coef0=0.0, degree=3,
gamma=0.001, kernel='rbf', max_iter=-1, probability=False,
random_state=None, shrinking=True, tol=0.001, verbose=False)
clf.fit(X_train, y_train)
clf.score(X_test, y_test)
y_pred = clf.predict(X_test)
The output is here:
precision recall f1-score support
-1.0 1.00 1.00 1.00 6
1.0 1.00 1.00 1.00 30
avg / total 1.00 1.00 1.00 36
Confusion Matrix:
[[ 6 0]
[ 0 30]]