Setting up ray (origin, direction) and triangle intersection(without glm)

Question

Edit3 : My problems were in completely different functions than i expected. ill let the code stay, maybe this helps someone :) (and dont forget to debug!).

Im trying to find the vector where a line intersects with a triangle.

Current state: Random intersections even if mouse is not at the floor and camera view dependend (lookat matrix)

Steps

1. Unproject mouse coordinations
2. Check line / triangle intersection

Unproject mouse coordinations

I checked the source of glm::unproject and gluUnproject and created this function.

   pixel::CVector3 pixel::CVector::unproject(
    CVector2 inPosition,
    pixel::CShape window,
    pixel::matrix4 projectionMatrix,
    pixel::matrix4 modelViewMatrix,
    float depth
    )
{
    // transformation of normalized coordinates
    CVector4 inVector;
    inVector.x = (2.0f * inPosition.x) / window.width - 1.0f;
    inVector.y = (2.0f * inPosition.y) / window.height - 1.0f;
    inVector.z = 2.0f * depth - 1.0f;
    inVector.w = 1.0f;

    // multiply inverted matrix with vector
    CVector4 rayWorld = pixel::CVector::multMat4Vec4(pixel::CMatrix::invertMatrix(projectionMatrix * modelViewMatrix), inVector);

    CVector3 result;
    result.x = rayWorld.x / rayWorld.w;
    result.y = rayWorld.y / rayWorld.w;
    result.z = rayWorld.z / rayWorld.w;



    return result;
}

Checking intersection

pixel::CVector3 pixel::Ray::intersection(
    Ray ray,
    pixel::CVector3 v0,
    pixel::CVector3 v1,
    pixel::CVector3 v2
    )
{
    // compute normal
    CVector3 a, b, n;
    a = v1 - v0;
    b = v2 - v0;

    n = ray.direction.cross(b);

    // find determinant
    float det = a.dot(n);

    if (det < 0.000001f)
    {
        std::cout << "Ray intersecting with backface triangles \n";
        return pixel::CVector::vector3(0.0f, 0.0f, 0.0f);
    }
    det = 1.0f / det;

    // calculate distance from vertex0 to ray origin
    CVector3 s = ray.origin - v0;
    float u = det * s.dot(n);

    if (u < -0.000001f || u > 1.f + 0.000001f)
    {
        std::cout << "U: Intersection outside of the triangle!\n";
        return pixel::CVector::vector3(0.0f, 0.0f, 0.0f);
    }

    CVector3 r = s.cross(a);
    float v = det * ray.direction.dot(r);
    if (v < -0.000001f || u + v > 1.f + 0.000001f)
    {
        std::cout << "V/U: Intersection outside of triangle!\n";
        return pixel::CVector::vector3(0.0f, 0.0f, 0.0f);
    }

    // distance from ray to triangle
    det = det *  b.dot(r);

    std::cout << "T: " << det << "\n";

    CVector3 endPosition;
    endPosition.x = ray.origin.x + (ray.direction.x * det);
    endPosition.y = ray.origin.y + (ray.direction.y * det);
    endPosition.z = ray.origin.z + (ray.direction.z * det);

    return endPosition;
}

Usage

    if (event.button.button == SDL_BUTTON_RIGHT)
            {

                camera->setCameraActive();
                float mx = event.motion.x;
                float my = window->info.height - event.motion.y;
                // ray casting
                pixel::Ray ray;

                std::cout << "\n\n";

                    // near
                pixel::CVector3 rayNear = pixel::CVector::unproject(
                    pixel::CVector::vector2(mx, my),
                    pixel::CVector::shape2(window->info.internalWidth, window->info.internalHeight),
                    camera->camInfo.currentProjection,
                    camera->camInfo.currentView,
                    1.0f
                    );
                // far
                pixel::CVector3 rayFar = pixel::CVector::unproject(
                    pixel::CVector::vector2(mx, my),
                    pixel::CVector::shape2(window->info.internalWidth, window->info.internalHeight),
                    camera->camInfo.currentProjection,
                    camera->camInfo.currentView,
                    0.0f
                    );


                // normalized direction results in the same behavior
                ray.origin = cameraPosition;

                ray.direction = pixel::CVector::normalize(rayFar- rayNear);

                std::cout << "Raycast \n";
                std::cout << "Mouse Position: " << mx << " - " << my << "\n";
                std::cout << "Camera Position: " << ray.origin.x << " - " << ray.origin.y << " - " << ray.origin.z << "\n";
                std::cout << "Ray direction: " << ray.direction.x << " - " << ray.direction.y << " - " << ray.direction.z << "\n";


                pixel::CVector3 vertOne = pixel::CVector::vector3(0.0f, 0.0f, -300.0f);
                pixel::CVector3 vertTwo = pixel::CVector::vector3(0.0f, 0.0f, 0.0f);
                pixel::CVector3 vertThree = pixel::CVector::vector3(300.0f, 0.0f, 0.0f);
                pixel::CVector3 vertFour = pixel::CVector::vector3(300.0f, 0.0f, -300.0f);


                pixel::CVector3 rayHit = pixel::Ray::intersection(ray, vertOne, vertTwo, vertThree);
                pixel::CVector3 rayHit2 = pixel::Ray::intersection(ray, vertThree, vertFour, vertOne);
                std::cout << "Ray hit: " << rayHit.x << " - " << rayHit.y << " - " << rayHit.z << "\n";
                std::cout << "Ray hit: " << rayHit2.x << " - " << rayHit2.y << " - " << rayHit2.z << "\n";
                std::cout << "--------------------\n";
                towerHouse->modelMatrix = pixel::CMatrix::translateMatrix(rayHit);

Output

As ive never used glm::unproject or gluUnproject, i dont know how the normal output should look like, but im getting results like:

Ray direction: 0.109035 -0.0380502 0.0114562

Doesnt look right to me, but checking my code against other sources (mentioned above), i dont see the mistake/s.

Ray intersection works in some special cases (camera rotation) and even then i get intersections even if i dont click on the floor. Same goes with intersection output varying from backface hits to outside of the triangle.

All those errors look like the main source of problem is the unprojection.

Any hints in the right direction?

Answer 1

This is nowhere close to an answer to this question, but this is too complicated to explain in comments or chat.

First of all:

            // near
            pixel::CVector3 rayNear = pixel::CVector::raycast(
                pixel::CVector::vector2(mx, my),
                pixel::CVector::shape2(window->info.internalWidth, window->info.internalHeight),
                camera->camInfo.currentProjection,
                camera->camInfo.currentView,
                1.0f // WRONG
                );
            // far
            pixel::CVector3 rayFar = pixel::CVector::raycast(
                pixel::CVector::vector2(mx, my),
                pixel::CVector::shape2(window->info.internalWidth, window->info.internalHeight),
                camera->camInfo.currentProjection,
                camera->camInfo.currentView,
                0.0f // WRONG
                );

Near is 0.0 in window-space, and far is 1.0 (depends on the depth range, but if you changed the depth range you should already know this).

In your ray cast function, you have:

CVector3 result;
result.x = rayWorld.x / rayWorld.w;
result.y = rayWorld.y / rayWorld.w;
result.z = rayWorld.z / rayWorld.w;

There is a chance that w == 0.0, and the result is not yet a ray at this time... it is a position in object-space (not world). Generally you are always going to be working with well-behaved matrices, but if you ever look at a formal implementation of UnProject (...) you will notice that they handle the case where w == 0.0 with a special return value or by setting a status flag.

            pixel::CVector3 vertOne = pixel::CVector::vector3(0.0f, 0.0f, -300.0f);
            pixel::CVector3 vertTwo = pixel::CVector::vector3(0.0f, 0.0f, 0.0f);
            pixel::CVector3 vertThree = pixel::CVector::vector3(300.0f, 0.0f, 0.0f);
            pixel::CVector3 vertFour = pixel::CVector::vector3(300.0f, 0.0f, -300.0f);

What coordinate space are these verts in? Presumably object-space, which means if you are casting a ray from your camera's eye point (defined in world-space) that passes through a point on your far plane, and try to test for intersection against a triangle in object-space more often than not you will miss. This is because the origin, scale and rotation for each of these spaces may differ. You need to transform those points into world-space (your original code had a floor->modelMatrix that would work well for this purpose) before you try this test.

Answer 2

I tracked down the problem and fixed bugs. i had wrong matrix*matrix and matrix*vector multiplications operators.