Is it vital to use weighted metrics with weighted loss functions?

Question

When one has an imbalanced machine learning problem, let's say a binary classification problem where class 0 is a majority class and class 1 is a minority class. Here class 1 is the most important class for you. Now your dataset is composed of 100 samples: 70 samples from class 0 and 30 samples from class 1.

In this case, usually, one defines and uses a weighted binary cross-entropy loss function to calculate the loss.

classweights = torch.FloarTensor([1.000, 1.300])
weighted_loss = torch.nn.BCELoss(weight = classweights)

For calculation of metrics like F1-score, is it required that these weights are used?

sample_classweights = sklearn.utils.class_weight.compute_sample_weight(class_weight, y_true)
classification_report = sklearn.metrics.classification_report(y_true, y_pred, sample_weight=sample_classweights)

Any hint or help is appreciated.

Answer 1

Calculation of loss and the calculation of metrics on the test set are two different entities. Usually, the weighted loss function is used to weight one of the class (you can use higher weights to the important class in balance or unbalanced class distribution).

For metrics like F1, its always safer to use multiple measures in unbalance class distribution. It would be good to keep a check on macro as well as weighted-F1.

For example: a confusion matrix of the following order:

predicted labels:     0   1
true label 0:      | 70 | 0 |
true label 1:      | 30 | 0 |

Has macro-F1: 0.41 and weighted-F1: 0.58

Further, Aconfusion matrix of the following order:

predicted labels:     0   1
true label 0:      | 99 | 0 |
true label 1:      | 1  | 0 |

Has macro-F1: 0.50 and weighted-F1: 0.99

Both these cases are unbalance where 70 and 90 samples belong to class 0 in 1st and 2nd case respectively. However, when the dataset is too unbalanced, and model predictions are biased towards majority class, then weighted F1 becomes higher.