Decision Level Fusion of SVR outputs

Question

I have two sets of features predicting the same outputs. But instead of training everything at once, I would like to train them separately and fuse the decisions. In SVM classification, we can take the probability values for the classes which can be used to train another SVM. But in SVR, how can we do this?

Any ideas?

Thanks :)

Answer 1

There are a couple of choices here . The two most popular ones would be:

ONE)

Build the two models and simply average the results.

It tends to work well in practice.

TWO)

You could do it in a very similar fashion as when you have probabilities. The problem is, you need to control for over fitting .What I mean is that it is "dangerous" to produce a score with one set of features and apply to another where the labels are exactly the same as before (even if the new features are different). This is because the new applied score was trained on these labels and therefore over fits in it (hyper-performs).

Normally you use a Cross-validation

In your case you have

1. train_set_1 with X1 features and label Y
2. train_set_2 with X2 features and same label Y

Some psedo code:

randomly split 50-50 both train_set_1 and train_set_2 at exactly the same points along with the Y (output array)

so now you have:

a.train_set_1 (50% of training_set_1)
b.train_set_1 (the rest of 50% of training_set_1)
a.train_set_2 (50% of training_set_2)
b.train_set_2 (the rest of 50% of training_set_2)
a.Y (50% of the output array that corresponds to the same sets as a.train_set_1 and a.train_set_2)
b.Y (50% of the output array that corresponds to the same sets as b.train_set_1 and b.train_set_2)

here is the key part

Build a svr with a.train_set_1 (that contains X1 features) and output a.Y and

Apply that model's prediction as a feature to b.train_set_2 .

By this I mean, you score the b.train_set_2 base on your first model. Then you take this score and paste it next to your a.train_set_2 .So now this set will have X2 features + 1 more feature, the score produced by the first model.

Then build your final model on b.train_set_2 and b.Y 

The new model , although uses the score produced from training_set_1, still it does so in an unbiased way , since the later was never trained on these labels!

You might also find this paper quite useful