Firstly, there is more than one way to form an orthonormal basis for a space. (For example [1 0; 0 1]
and 1/sqrt(2) * [ 1 -1; 1 1 ]
both describe the same 2D Euclidean space). So we wouldn't necessarily expect two alternative implementations to pick the same basis set.
If we take the right-hand three columns in each case, we learn the following:
> Vmat = Vmat(:,4:end);
> Veig = Veig(:,4:end);
> Vmat' * V_mat
ans =
1.0000e+00 8.8800e-06 -1.4120e-05
8.8800e-06 9.9999e-01 -5.1830e-05
-1.4120e-05 -5.1830e-05 1.0000e+00
> Veig' * Veig
ans =
1.0001e+00 -1.4050e-05 2.4200e-06
-1.4050e-05 1.0001e+00 -4.8310e-05
2.4200e-06 -4.8310e-05 1.0000e+00
> A * Vmat
ans =
7.7612e-17 7.8916e-17 0.0000e+00
-4.1193e-17 4.8139e-17 0.0000e+00
6.6136e-18 -6.0715e-18 1.1102e-16
> A * Veig
ans =
-1.2030e-05 1.1000e-05 -6.0000e-07
-4.8600e-06 3.8750e-05 1.5490e-05
-3.4400e-06 -4.5210e-05 -3.6090e-05
So these are both orthonormal basis sets, and they're both basically null spaces. However, the error level in the Eigen case appears to correspond to the fact that it was done in single-precision. Try it again in double-precision, and see how the results compare this time (I'm not claiming this will definitely help, merely that this is one obvious difference with Matlab.)