How do you view the labeling of the test set with GenericAcrfTui from the command line?

Question

I am training and testing data using Mallet's GenericAcrfTui. So I am using the Graphical Models in Mallet (GRMM) to do CRF training. I have created features for both my training set and my test set and was hoping to run GenericAcrfTui from the command line. When I run it, I get back the F-score and accuracy measures for each of my labels, but I do not get back the actual labeling for each row. I do something like this:

java -cp $GRMM/class:$GRMM/lib/mallet-deps.jar:$GRMM/lib/grmm-deps.jar \
edu.umass.cs.mallet.grmm.learning.GenericAcrfTui \ 
--training $GRMM/data/grmm/conll2000.train1k.txt \
--testing  $GRMM/data/grmm/conll2000.test1k.txt \
--model-file tmpls.txt > stdout.txt 2> stderr.txt

which is the example given in the Mallet example. There does not appear to be a switch though for getting the labels. How does one do it?

Answer 1

The idea is to write your own Evaluator (that is subclass ACRFEvaluator). There are two methods you must write, evaluate and test. The evaluate method can be copied code for another Evaluator in the source code, no big deal. It is the test method where you add the lines to perform the task.

In general, the output of command line is split between stdout and the logger used. The results for each label (f-score and accuracy) end up in the logger output so that may be where we put the labels, otherwise perhaps a separate file for the labels, your choice. So here is a code example for a test method. I will output to stdout and you adjust to your tastes.

@Override
public void test(InstanceList gold, List returned, String description) {
    for (Iterator it = evals.iterator(); it.hasNext(); ) {
        ACRFEvaluator eval = (ACRFEvaluator) it.next();
        eval.test(gold, returned, description);
        int rows = returned.size();
        for (int index = 0; index < rows; index++)                
            System.out.println("PREDICTED: " + returned.get(index).toString() + " vs. " + gold.get(index).toString());
        }
    }
}

For completeness, the class declaration is:

public class MyEvaluator extends ACRFEvaluator {
    // body goes here
}

And the switch from the command line would look like

--eval "new MyEvaluator()"