As a simple approach, you can train an additional classifier to determine if your feature is a digit or not. Use non-digit images as positive examples and the other classes' positives (i.e. images of digits 0-9) as the negative samples of this classifier. You'll need a huge amount of non-digit images to make it work, and also it's recommendable to use strategies as the selection of hard negatives: negative samples classified as "false positives" after the first training stage, which are used to re-train the classifier.
Hope that it helps!