SVMlight - what is the purpose of finding the weight vector?

Question

The SVMlight site (in the FAQ) offers a script that computes the weight vector of a hyperlane. It says it "computes the weighted sum of the support vectors".

What does this mean? That is, what does the script do and what is the meaning and use of this weight vector?

Thanks in advance!

Answer 1

In linear case, the hyperplane can be always defined with d+1 numbers, where d is the dimension of the input space, while the number of actual support vectors may be much larger. By computing this hyperplane (lets call it w) you get more compact model, which can be then used to perform a classification:

cl(x) = sgn(w'x + b)

where w' is a transpositon of w

Things get much more tricky in the kernelized version, as w is in the form of the feature space projection, which may be unknown (or to expensive to compute) so one cannot get an equation of such an object (as it is no longer a hyperplane in the input space, but rather - a hyperplane in very rich feature space).

Answer 2

"Support vectors are the elements of the training set that would change the position of the dividing hyperplane if removed." The weights represent this hyperplane by providing the coordinates of a vector that is orthogonal to the hyperplane. "Computes the weighted sum of the support vectors" mathematically means sign(w'*x +b), when x is the support vectors and w' is the transpose of weight vectors, the value of w'x+b is 0 and it represents the decision boundary. When a new x reaches, the sign(w'x+b) will determine which class it belongs to.

For those x in the training sample that have the weight of 0, it means the sample does not contribute to the hyperplane, and including that x as a support vector will either increase the classification error, or decrease the margin.

Here is a reference tutorial with plenty of figures for more details.