ray casting from mouse with opengl 2

Question

I'm attempting to do ray casting on mouse click with the eventual goal of finding the collision point with a plane. However I'm unable to create the ray. The world is rendered using a frustum and another matrix I'm using as a camera, in the order of frustum * camera * vertex_position. With the top left of the screen as 0,0 I'm able to get the X,Y of the click in pixels. I then use the below code to convert this to the ray:

  float x = (2.0f * x_screen_position) / width - 1.0f;
  float y = 1.0f - (2.0f * y_screen_position) / height;
  Vector4 screen_click = Vector4 (x, y, 1.0f, 1.0f);
  Vector4 ray_origin_world = get_camera_matrix() * screen_click;

  Vector4 tmp = (inverse(get_view_frustum()) * screen_click;
  tmp = get_camera_matrix() * tmp;
  Vector4 ray_direction = normalize(tmp);

view_frustum matrix:

Matrix4 view_frustum(float angle_of_view, float aspect_ratio, float z_near, float z_far) {
    return Matrix4(
        Vector4(1.0/tan(angle_of_view), 0.0, 0.0, 0.0),
        Vector4(0.0, aspect_ratio/tan(angle_of_view), 0.0, 0.0),
        Vector4(0.0, 0.0, (z_far+z_near)/(z_far-z_near), 1.0),
        Vector4(0.0, 0.0, -2.0*z_far*z_near/(z_far-z_near), 0.0)
    );
}

When the "camera" matrix is at 0,0,0 this gives the expected results however once I change to a fixed camera position in another location the results returned are not correct at all. The fixed "camera" matrix:

Matrix4(
    Vector4(1.0, 0.0, 0.0, 0.0),
    Vector4(0.0, 0.70710678118, -0.70710678118, 0.000),
    Vector4(0.0, 0.70710678118, 0.70710678118, 0.0),
    Vector4(0.0, 8.0, 20.0, 1.000)
  );

Because many examples I have found online do not implement a camera in such a way I am unable to found much information to help in this case. Can anyone offer any insight into this or point me in a better direction?

Answer 1

Vector4 tmp = (inverse(get_view_frustum() * get_camera_matrix()) * screen_click; //take the inverse of the camera matrix as well
tmp /= tmp.w; //homogeneous coordinate "normalize" (different to typical normalization), needed with perspective projection or non-linear depth
Vector3 ray_direction = normalize(Vector3(tmp.x, tmp.y, tmp.z)); //make sure to normalize just the direction without w

---

[EDIT]

A more lengthy and similar post is here: https://stackoverflow.com/a/20143963/1888983

If you only have matrices, a start point and point in the ray direction should be used. It's common to use points on the near and far plane for this (an advantage is if you only want the ray to intersect things that are visible). That is,

(x, y, -1, 1) to (x, y, 1, 1)

These points are in normalized device coordinates (NDC, a -1 to 1 cube that is your viewing volume). All you need to do is move both points all the way to world space and normalize...

ndcPoint4 = /* from above */;
eyespacePoint4 = inverseProjectionMatrix * ndcPoint4;
worldSpacePoint4 = inverseCameraMatrix * eyespacePoint4;
worldSpacePoint3 = worldSpacePoint4.xyz / worldSpacePoint4.w;

//alternatively, with combined matrices
worldToClipMatrix = projectionMatrix * cameraMatrix; //called "clip" space before normalization
clipToWorldMatrix = inverse(worldToClipMatrix);
worldSpacePoint4 = clipToWorldMatrix * ndcPoint4;
worldSpacePoint3 = worldSpacePoint4.xyz / worldSpacePoint4.w;

//then for the ray, after transforming both start/end points
rayStart = worldPointOnNearPlane;
rayEnd = worldPointOnFarPlane;
rayDir = rayEnd - rayStart;

If you have the camera's world space position, you can drop either start or end point since all rays pass through the camera's origin.