back-projecting a 2D point to 3D Plucker line

Question

I'm trying to build a tracker (hand tracking personal project), and for this reason I need to back-project a 2d point to a 3d line, using Plucker coordinates. (like ray tracing)

As an input, I have the 2d coordinates of a point and the projection matrix.

The information on the web about plucker coordinates give an overview of why they helpful, but there's no paper describing analytically the above mentioned procedure. (they just mention that they back-project to a plucker line, without any further description)

Can somebody please pinpoint me to the right direction?

Answer 1

Apparently there's no magic behind this, I was looking for a formula/theorem resulting immediately from my 'inputs' to plucker coordinates, while there's no such thing.

As inputs, we have

• 2D coordinates of a (projected) point
• the projection matrix

Using these two inputs, we can back-project this 2d point to a ray (3D line). All 3D points of this 3D line are projected to the same 2D point. The ray by default passes through the camera center (or projection center,etc).

For necessary equations, see

• Hartley/Zisserman book @ page 162 @ equation 6.14, OR
• Equation 23 -> http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/FUSIELLO4/tutorial.html

The general idea is that, in order to define a line, you need 2 points. We choose to find (with equations from the above mentioned sources)

• The camera center (all projection rays pass by default through this point)
• A point of the ray @ infinity (this is nice, because a point at infinity is also the direction vector of the line -> this is needed later-on for the plucker line representation)

(we could have found the camera center and another arbitrary point, but then we would need an extra step to find the direction of the line, by subtracting the coordinates of these two points.)

To sum up, we have found

• The camera center (p)
• The direction (d) of the line (point @ infinity) (Points at infinity are equivalent to directions)

These are enough to represent a line, but this representation is not optimal when we have to compute for example distances of 3D points to this line in our algorithm. This is why, after finding this representation (no magic to give us plucker lines immediately), we change our line-representation to plucker-line-representation

Plucker line is just another representation of the line that needs:

• The direction of the line (we already have it!!! -> d -> point at infinity)

• The 'moment' (m) of the line, which is easy to compute from the previous representation:

  m=p^d (^ -> cross product)

I hope this clears things for anyone that will need this in the future, I think it's a very easy thing, but in the beginning things might not be so apparent.

For a practical scenario, why one would use this plucker-line-representation, please check

• http://files.is.tue.mpg.de/dtzionas/Hand-Object-Capture/IJCV_Hand_Object_Capture.pdf
• https://pages.iai.uni-bonn.de/gall_juergen/download/jgall_tpami09.pdf

Answer 2

For future reference using Matlab / Octave syntax!

The join of two points in Plücker coordinates can be expressed as follows

% line = point join point
function L=join(A, B)
L=[
        A(1)*B(2)-A(2)*B(1);
        A(1)*B(3)-A(3)*B(1);
        A(1)*B(4)-A(4)*B(1);
        A(2)*B(3)-A(3)*B(2);
        A(2)*B(4)-A(4)*B(2);
        A(3)*B(4)-A(4)*B(3)
];
end % function

These are the 6 distinct values from the anti-symmetric matrix

Lx=B*A'-A*B'

A point on the backprojection ray can be found

X=pinv(P)*x

where

x=[u v 1]'

is the image point at pixel location (u,v) and

pinv(P)

the pseudoinverse of the projection matrix.

The camera center can be found as the null space of the projection matrix

C=null(P);
C=C/C(4)

The Plücker coordinates of the backprojection ray are thus

L=join(X,C)

For those interested in oriented projective geometry: If you normalize the projection matrix as follows

% Get length of principal ray
m3n=norm(P(3,1:3));
% Enforce positivity of determinant
if (det(P(:,1:3))<0)
    m3n=-m3n;
end % if
% Normalize
P=P/m3n;

Then the determinant of the left 3x3 matrix is positive (i.e. right handed systems) and L will point from C to X.

Update: The other day, a co-worker asked me for a matrix-form solution. If you multiply the following matrix with an image point [x0 x1 x2]' you get the Plücker coordinates directly:

A=[
  - p12*p23 + p13*p22, + p02*p23 - p03*p22, + p03*p12 - p02*p13  
  + p11*p23 - p13*p21, - p01*p23 + p03*p21, + p01*p13 - p03*p11  
  - p11*p22 + p12*p21, + p01*p22 - p02*p21, + p02*p11 - p01*p12  
  - p10*p23 + p13*p20, + p00*p23 - p03*p20, + p03*p10 - p00*p13  
  + p10*p22 - p12*p20, - p00*p22 + p02*p20, + p00*p12 - p02*p10  
  - p10*p21 + p11*p20, + p00*p21 - p01*p20, + p01*p10 - p00*p11  
]

The line direction is therefore

d =
     p01*p12*x2 - p02*p11*x2 - p01*p22*x1 + p02*p21*x1 + p11*p22*x0 - p12*p21*x0
     p02*p10*x2 - p00*p12*x2 + p00*p22*x1 - p02*p20*x1 - p10*p22*x0 + p12*p20*x0
     p00*p11*x2 - p01*p10*x2 - p00*p21*x1 + p01*p20*x1 + p10*p21*x0 - p11*p20*x0

and the line moment is:

m =
     p03*p10*x2 - p00*p13*x2 + p00*p23*x1 - p03*p20*x1 - p10*p23*x0 + p13*p20*x0
     p03*p11*x2 - p01*p13*x2 + p01*p23*x1 - p03*p21*x1 - p11*p23*x0 + p13*p21*x0
     p03*p12*x2 - p02*p13*x2 + p02*p23*x1 - p03*p22*x1 - p12*p23*x0 + p13*p22*x0

Obviously, one can pull out the [x2 x1 x0] from the expression to get matrix form. The following MATLAB symbolic code was used to generate this solution:

syms p00 p01 p02 p03 p10 p11 p12 p13 p20 p21 p22 p23 real
P=[ p00 p01 p02 p03 
    p10 p11 p12 p13 
    p20 p21 p22 p23 
]

% Some Image Point
syms x0 x1 x2 real 
x=[x0 x1 x2]'

% Source Position
C=null(P)

% Backprojection of x
Xtilde=pinv(P)*x

% Backprojection line (Plücker Matrix)
Lx=simplify(C*Xtilde' - Xtilde*C')

% It's homogeneous...
arbitrary_scale=(p00*p11*p22 - p00*p12*p21 - p01*p10*p22 + p01*p12*p20 + p02*p10*p21 - p02*p11*p20)
Lx=Lx*arbitrary_scale

% Plücker Coordinates:
L=[Lx(1,2) Lx(1,3) Lx(1,4) Lx(2,3) Lx(2,4) Lx(3,4)];
% Direction
d=[-L(3) -L(5) -L(6)]';
% Moment
m=[L(4) -L(2) L(1)]';


% In matrix form
A=[
  - p12*p23 + p13*p22, + p02*p23 - p03*p22, + p03*p12 - p02*p13  
  + p11*p23 - p13*p21, - p01*p23 + p03*p21, + p01*p13 - p03*p11  
  - p11*p22 + p12*p21, + p01*p22 - p02*p21, + p02*p11 - p01*p12  
  - p10*p23 + p13*p20, + p00*p23 - p03*p20, + p03*p10 - p00*p13  
  + p10*p22 - p12*p20, - p00*p22 + p02*p20, + p00*p12 - p02*p10  
  - p10*p21 + p11*p20, + p00*p21 - p01*p20, + p01*p10 - p00*p11  
]

% Verification: (should be zero)
simplify(A*x - L')

Answer 3

I'm posting this just for the sake of completeness, based on a combination of some stuff in the paper cited in the OP's accepted answer, the answer by @André Aichert and the descriptions in p493 of [1].

The following is a minimally working MATLAB example for constructing the Plucker line passing through two points A and B and computing its distance to point C.

A = [0 0 0]';
B = [0 0 5]';
C = [1 1 0]';

L = pluckerline(A,B);

distance = compute_plucker_distance(C, L) % Will output 1.4142

%%-------------------------------------------------------------------------

% Comptes the Plucker line passing through points A and B
function L = pluckerline(A, B)
    l = (B - A) / norm(B - A);
    m = cross(A, l);

    L = [l ; m];
end

%%-------------------------------------------------------------------------

% Comptes the distance between the point P and Plucker line L
function distance = compute_plucker_distance(P, L)
    l = L(1:3);
    m = L(4:end);

    distance = norm(cross(P, l) - m);
end

[1] Sommer, Gerald, ed. Geometric computing with Clifford algebras: theoretical foundations and applications in computer vision and robotics. Springer Science & Business Media, 2013.