understanding an Opengl shader code - cubemap?

Question

I found this code on internet http://rioki.org/2013/03/07/glsl-skybox.html for cubemap enviromental texture(actually rendering a skybox). But i do not understand it why it works.

void main()
{
    mat4 r = gl_ModelViewMatrix;
    r[3][0] = 0.0;
    r[3][1] = 0.0;
    r[3][2] = 0.0;

    vec4 v = inverse(r) * inverse(gl_ProjectionMatrix) * gl_Vertex;

    gl_TexCoord[0] = v; 
    gl_Position    = gl_Vertex;
} 

So gl_Vertex is in world coordinates but what do we get by multiplying that by inverse of projection matrix and then modelview matrix?

this is the code I use to draw my skybox

void SkyBoxDraw(void)
{
GLfloat SkyRad = 1.0f;

glUseProgramObjectARB(glsl_program_skybox);
glDepthMask(0);
glDisable(GL_DEPTH_TEST);
// Cull backs of polygons
glCullFace(GL_BACK);
glEnable(GL_CULL_FACE);
glEnable(GL_TEXTURE_CUBE_MAP);

glBegin(GL_QUADS);
    //////////////////////////////////////////////
    // Negative X
    glTexCoord3f(-1.0f, -1.0f, 1.0f);
    glVertex3f(-SkyRad, -SkyRad, SkyRad);

    glTexCoord3f(-1.0f, -1.0f, -1.0f);
    glVertex3f(-SkyRad, -SkyRad, -SkyRad);

    glTexCoord3f(-1.0f, 1.0f, -1.0f);
    glVertex3f(-SkyRad, SkyRad, -SkyRad);

    glTexCoord3f(-1.0f, 1.0f, 1.0f);
    glVertex3f(-SkyRad, SkyRad, SkyRad);
    ......
    ......
glEnd();

glDepthMask(1);
glDisable(GL_CULL_FACE);
glEnable(GL_DEPTH_TEST);
glDisable(GL_TEXTURE_CUBE_MAP);

glUseProgramObjectARB(0);

}

Answer 1

So gl_Vertex is in world coordinates ...

No no no, gl_Vertex is in object/model-space unless I see some code elsewhere (e.g. how your vertex position is calculated in the actual non-shader portion of your program) that indicates otherwise :) In OpenGL we skip from object-space to eye/view/camera-space when we multiply by the combined Model*View matrix. As you can see, there are lots of names for the same coordinate spaces, but object-space definitely is not a synonym for world-space. Setting r3 to < 0, 0, 0, 1 > basically re-positions the camera's origin without affecting direction, which is useful when all you want to know is the direction for a cubemap lookup.

That is, in a nutshell, what you want when using cubemaps. Just a simple direction vector. The fact that textureCube (...) takes a 3D vector instead of 4D is an immediate hint that it is looking for a direction instead of position. Position vectors have a 4th component, direction do not. So, technically if you wanted to port this shader to modern OpenGL you would probably use an out vec3 and swizzle .xyz off of v, since v.w is unnecessary.

... but what do we get by multiplying that by inverse of projection matrix and then modelview matrix?

You are basically undoing projection when you multiply by the inverse of these matrices. The only way this shader makes sense is if the coordinates you are passing for your vertices are defined in clip-space. So instead of going from object-space through the GL pipeline and winding up in screen-space at the end you want the reverse of that, only since the viewport is not involved in your shader we cannot be dealing with screen-space. A little bit more information on how your vertex positions are calculated should clear this up.