OpenGL Asymmetric Frustum for Desktop VR

Question

I am making an OpenGL c++ application that tracks the users location in relation to the screen and then updates the rendered scene to the perspective of the user. This is know as "desktop VR" or you can think of the screen as a diorama or fish tank. I am rather new to OpenGL and have only defined a very simple scene thus far, just a cube, and it is initially rendered correctly.

The problem is that when I start moving and want to rerender the cube scene, the projection plane seems translated and I don't see what I think I should. I want this plane fixed. If I were writing a ray tracer, my window would always be fixed, but my eye is allowed to wander. Can someone please explain to me how I can achieve the effect I desire (pinning the viewing window) while having my camera/eye wander at a non-origin coordinate?

All of the examples I find demand that the camera/eye be at the origin, but this is not conceptually convenient for me. Also, because this is a "fish tank", I am setting my d_near to be the xy-plane, where z = 0.

In screen/world space, I assign the center of the screen to (0,0,0) and its 4 corners to: TL(-44.25, 25, 0) TR( 44.25, 25, 0) BR( 44.25,-25, 0) BL(-44.25,-25, 0) These values are in cm for a 16x9 display.

I then calculate the user's eye (actually a web cam on my face) using POSIT to be usually somewhere in the range of (+/-40, +/-40, 40-250). My POSIT method is accurate.

I am defining my own matrices for the perspective and viewing transforms and using shaders.

I initialize as follows:

float right = 44.25;
float left = -44.25;
float top = 25.00;
float bottom = -25.00; 

vec3 eye = vec3(0.0, 0.0, 100.0);
vec3 view_dir = vec3(0.0, 0.0, -1.0);
vec3 up = vec3(0.0, 1.0, 0.0);
vec3 n = normalize(-view_dir);
vec3 u = normalize(cross(up, n)); 
vec3 v = normalize(cross(n, u));

float d_x = -(dot(eye, u));
float d_y = -(dot(eye, v));
float d_z = -(dot(eye, n));

float d_near = eye.z;
float d_far = d_near + 50;

// perspective transform matrix
mat4 P = mat4((2.0*d_near)/(right-left ), 0, (right+left)/(right-left), 0, 
            0, (2.0*d_near)/(top-bottom), (top+bottom)/(top-bottom), 0,
            0, 0, -(d_far+d_near)/(d_far-d_near), -(2.0*d_far*d_near)/(d_far-d_near),
            0, 0, -1.0, 0);

// viewing transform matrix
mat4 V = mat4(u.x, u.y, u.z, d_x,
              v.x, v.y, v.z, d_y,
              n.x, n.y, n.z, d_z,
              0.0, 0.0, 0.0, 1.0);

mat4 MV = C * V;
//MV = V;

From what I gather looking on the web, my view_dir and up are to remain fixed. This means that I need only to update d_near and d_far as well as d_x, d_y, and d_y? I do this in my glutIdleFunc( idle );

void idle (void) {  

    hBuffer->getFrame(hFrame);
    if (hFrame->goodH && hFrame->timeStamp != timeStamp) {
        timeStamp = hFrame->timeStamp;
        std::cout << "(" << hFrame->eye.x << ", " <<
                    hFrame->eye.y << ", " <<
                    hFrame->eye.z << ") \n";

        eye = vec3(hFrame->eye.x, hFrame->eye.y, hFrame->eye.z);

        d_near = eye.z;
        d_far = eye.z + 50;

        P = mat4((2.0*d_near)/(right-left), 0, (right+left)/(right-left), 0, 
                 0, (2.0*d_near)/(top-bottom), (top+bottom)/(top-bottom), 0,
                 0, 0, -(d_far+d_near)/(d_far-d_near), -(2.0*d_far*d_near)/(d_far-d_near),
                 0, 0, -1.0, 0);

        d_x = -(dot(eye, u));
        d_y = -(dot(eye, v));
        d_z = -(dot(eye, n));

        C = mat4(1.0, 0.0, 0.0, eye.x,
                 0.0, 1.0, 0.0, eye.y,
                 0.0, 0.0, 1.0, 0.0,
                 0.0, 0.0, 0.0, 1.0);

        V = mat4(u.x, u.y, u.z, d_x,
                 v.x, v.y, v.z, d_y,
                 n.x, n.y, n.z, d_z,
                 0.0, 0.0, 0.0, 1.0);

        MV = C * V;
        //MV = V;

        glutPostRedisplay();
    }
}

Here is my shader code:

#version 150

uniform mat4 MV;
uniform mat4 P;
in vec4 vPosition;
in vec4 vColor;
out vec4 color;

void 
main() 
{ 
    gl_Position = P * MV * vPosition;
    color = vColor;
}

Ok, I made some changes to my code, but without success. When I use V in place of MV in the vertex shader, everything looks as I want it to, the perspective is correct and the objects are the right size, however, the scene is translated by the displacement of the camera.

When using C and V to obtain MV, my scene is rendered from the perspective of an observer straight on and the rendered scene fills the window as it should, but the perspective of the eye/camera is lost.

Really, what I want is to translate the 2D pixels, the projection plane, by the appropriate x and y values of the eye/camera, so as to keep the center of the object (whose xy center is (0,0)) in the center of the rendered image. I am guided by the examples in the textbook "Interactive Computer Graphics: A Top-Down Approach with Shader-Based OpenGL (6th Edition)". Using the files paired with the book freely available on the web, I am continuing with the row major approach.

The following images are taken when not using the matrix C to create MV. When I do use C to create MV, all scenes look like the first image below. I desire no translation in z and so I leave that as 0.

Because the projection plane and my camera plane are parallel, the conversion from one to the other coordinate system is simply a translation and inv(T) ~ -T.

Here is my image for the eye at (0,0,50):

Here is my image for the eye at (56,-16,50):

Answer 1

The solution is to update d_x, d_y, and d_z, accounting for the new eye position, but to never change u, v, or n. In addition, one must update the matrix P with new values for left, right, top and bottom, as they relate to the new position of the camera/eye.

I initialize with this:

float screen_right = 44.25;
float screen_left = -44.25;
float screen_top = 25.00;
float screen_bottom = -25.00; 
float screen_front = 0.0;
float screen_back = -150;

And now my idle function looks like this, note the calculations for top, bottom, right, and left:

void idle (void) {  

hBuffer->getFrame(&hFrame);
if (hFrame.goodH && hFrame.timeStamp != timeStamp) {
    timeStamp = hFrame.timeStamp;
    //std::cout << "(" << hFrame.eye.x << ", " <<
    //                  hFrame.eye.y << ", " <<
    //                  hFrame.eye.z << ") \n";

    eye = vec3(hFrame.eye.x, hFrame.eye.y, hFrame.eye.z);

    d_near = eye.z;
    d_far = eye.z + abs(screen_back) + 1;

    float top = screen_top - eye.y;
    float bottom = top - 2*screen_top;
    float right = screen_right - eye.x;
    float left = right - 2*screen_right;

    P = mat4((2.0 * d_near)/(right - left ), 0.0, (right + left)/(right - left), 0.0, 
             0.0, (2.0 * d_near)/(top - bottom), (top + bottom)/(top - bottom), 0.0,
             0.0, 0.0, -(d_far + d_near)/(d_far - d_near), -(2.0 * d_far * d_near)/(d_far - d_near),
             0.0, 0.0, -1.0, 0.0);

    d_x = -(dot(eye, u));
    d_y = -(dot(eye, v));
    d_z = -(dot(eye, n));

    V = mat4(u.x, u.y, u.z, d_x,
             v.x, v.y, v.z, d_y,
             n.x, n.y, n.z, d_z,
             0.0, 0.0, 0.0, 1.0);

    glutPostRedisplay();
}

}

This redefinition of the perspective matrix keeps the rendered image from translating. I still have camera capture and screen grabbing synchronization issues, but I am able to create the images like following updated in real-time for the position of the user:

Answer 2

All of the examples I find demand that the camera/eye be at the origin

This is not a demand. OpenGL doesn't define a camera, it all boils down to transformations. Those are usually called projection and modelview (model and view transform combined). The camera, i.e. view transform is simply the inverse of the positioning of the camera in world space. So say you built yourself a matrix for your camera C, then you'd premultiply it's inverse on the modelview.

mat4 MV = inv(C) * V

On a side note, your shader code is wrong:

gl_Position = P * ( (V * vPosition ) / vPosition.w );
                                    ^^^^^^^^^^^^^^

The homogenous divide is not to be done in the shader, as it is hardwired into the render pipeline.