Caret Binary Classification with RMSE

Question

Is there a way to get caret to use RMSE with a binary classification problem?

If you try to use metric = "RMSE" with a classification problem you will receive the message:

Error in train.default(x, y, weights = w, ...) :
    Metric RMSE not applicable for classification models

Which makes sense. But is there a way to define a custom metric? For example, if your outcome is 0 or 1, you can define the error as outcome - p where p is the probability predicted by the model.

EDIT ====================

To give this some context and for some reasoning behind wanting to use this measure, see 2.7.1 in An Experimental Analysis of Classifier Ensembles for Learning Drifting Concepts Over Time in Autonomous Outdoor Robot Navigation by Michael J. Procopio, or the paper on softclassval

Answer 1

I don't know why you would want to do this but you can make your own summary function:

library(caret)

set.seed(1)
dat <- twoClassSim(100)

foo <- function(data, lev = NULL, model = NULL) {
  probs <- data[, lev[1]]
  c(rmse = RMSE(pred = probs,
                obs = ifelse(data$obs == lev[1], 1, 0)))
}

ctrl <- trainControl(classProbs = TRUE,
                     summaryFunction = foo)
set.seed(2)
mod <- train(Class ~ ., data = dat,
             method = "lda",
             metric = "rmse",
             minimize = TRUE,
             trControl = ctrl)

Max

Answer 2

Y should be a factor. Use as.factor() on it before training the model.