Cross Validation - Weka API

Question

How can I make a classification model by 10-fold cross-validation using Weka API? I ask this, because each cross-validation's run a new classification model is created. Which classification model should I use in my test data?

Answer 1

10-fold cross validation is used to get an estimate of a classifier's accuracy should that classifier be constructed from all of the training data. It is used when it is felt that there is not enough data for an independent test set. This means that you should build a new model from all the training data when you go to predict future data. The result from 10-fold cross validation is a guess as to how well your new classifier should perform.

The following code shows an example of using Weka's cross-validation through the API, and then building a new model from the entirety of the training dataset.

    //Training instances are held in "originalTrain"

    Classifier c1 = new NaiveBayes();
    Evaluation eval = new Evaluation(originalTrain);
    eval.crossValidateModel(c1, originalTrain, 10, new Random(1));
    System.out.println("Estimated Accuracy: "+Double.toString(eval.pctCorrect()));

    //Train a new classifier
    Classifier c2 = new NaiveBayes();
    c2.buildClassifier(originalTrain)  //predict with this model

Answer 2

Perform cross-validation with:

// perform cross-validation             
    for (int n = 0; n < folds; n++) {
        //Evaluation eval = new Evaluation(randData);
        //get the folds       
        Instances train = randData.trainCV(folds, n);
        Instances test = randData.testCV(folds, n);       

        ArffSaver saver = new ArffSaver();
        saver.setInstances(train);
        System.out.println("No of folds done = " + (n+1));

    saver.setFile(new File("C:\\\\Users\\AmeerSameer\\Desktop\\mytrain.arff"));
        saver.writeBatch();
        //if(n==9)
        //{System.out.println("Training set generated after the final fold is");
        //System.out.println(train);}

        ArffSaver saver1 = new ArffSaver();
        saver1.setInstances(test);
        saver1.setFile(new File("C:\\Users\\AmeerSameer\\Desktop\\mytest.arff"));
        saver1.writeBatch();