2D Screen coordinate to 3D position Directx 9 / Box Select

Question

I am trying to implement box select in a 3d world. Basically, click, hold mouse, and then unpress mouse, get a box, and then box select. To start, I'm trying to figure out how to get the coordinates of the clicks in 3d.

I have raypicking, and that is not getting the right coordinate (gets origin and direction). It keeps returning the same origin no matter what X/Y for screen is (although the direction is different).

I've also tried:

D3DXVECTOR3 ori = D3DXVECTOR3(sx, sy, 0.0f);
D3DXVECTOR3 out;
D3DXVec3Unproject(&out, &ori, &viewPort, &projectionMat, &viewMat, &worldMat);

And it gets the same thing, the coordinates are very close to each other no matter what coordinates (and are wrong). It's almost like returning the eye, instead of the actual world coordinate.

How do I turn 2d Screen coordinates into 3d using directx 9c?

Answer 1

It turns out, I was handling the problem the wrong/opposite way. Turning 2D to 3D didn't make sense in the end. But as it turns out, converting the vertices from 3D to 2D, then seeing if inside the 2D box was the right answer!

Answer 2

This is called picking in Direct3D, to select a model in 3D space, you mainly need 3 steps:

1. Generate the picking ray
2. Transform the picking ray and the model you want to pick in the same space
3. Do a intersection test of the picking ray and the model

Generate the picking ray

When we click the mouse on the screen(say the point is s on the screen), the model is selected when the box project on the area surround the point s on the projection window. so, in order to generate the picking ray with the given screen coordinates (x, y), first we need to transform (x,y) to the projection window, this is can be done by the invert process of viewport transformation. another thing is the point on the projection window was scaled by the project matrix, so we should divide it by the scale factors. in DirectX, the camera always place at the origin, so the picking ray starts from the origin, and projection window is the near clip plane(z=1).this is what the code has done below.

Ray CalcPickingRay(LPDIRECT3DDEVICE9 Device, int screen_x, int screen_y) 
{
     float px = 0.0f;
     float py = 0.0f;

     // Get viewport
     D3DVIEWPORT9 vp;
     Device->GetViewport(&vp);

     // Get Projection matrix
     D3DXMATRIX proj;
     Device->GetTransform(D3DTS_PROJECTION, &proj);

     px = ((( 2.0f * screen_x) / vp.Width)  - 1.0f) / proj(0, 0);
     py = (((-2.0f * screen_y) / vp.Height) + 1.0f) / proj(1, 1);

     Ray ray;
     ray._origin    = D3DXVECTOR3(0.0f, 0.0f, 0.0f);
     ray._direction = D3DXVECTOR3(px, py, 1.0f);

     return ray;
}

Transform the picking ray and model into the same space.

We always obtain this by transform the picking ray to world space, simply get the invert of your view matrix, then apply the invert matrix to your pickig ray. // transform the ray from view space to world space

void TransformRay(Ray* ray, D3DXMATRIX* invertViewMatrix)
{
    // transform the ray's origin, w = 1.
    D3DXVec3TransformCoord(
        &ray->_origin,
        &ray->_origin,
        invertViewMatrix);

    // transform the ray's direction, w = 0.
    D3DXVec3TransformNormal(
        &ray->_direction,
        &ray->_direction,
        invertViewMatrix);

    // normalize the direction
    D3DXVec3Normalize(&ray->_direction, &ray->_direction);
}

Do intersection test

If everything above is well, you can do the intersection test now. this is a ray-box intersection, so you can use function D3DXboxBoundProbe. you can change the visual mode of you box to see if the picking was really work, for example, set the fill mode to solid or wire-frame if D3DXboxBoundProbe return TRUE.

You can perform the picking in response of WM_LBUTTONDOWN.

case WM_LBUTTONDOWN:
{
    // Get screen point
    int iMouseX = (short)LOWORD(lParam) ;
    int iMouseY = (short)HIWORD(lParam) ;

    // Calculate the picking ray
    Ray ray = CalcPickingRay(g_pd3dDevice, iMouseX, iMouseY) ;

    // transform the ray from view space to world space
    // get view matrix
    D3DXMATRIX view;
    g_pd3dDevice->GetTransform(D3DTS_VIEW, &view);

    // inverse it
    D3DXMATRIX viewInverse;
    D3DXMatrixInverse(&viewInverse, 0, &view);

    // apply on the ray
    TransformRay(&ray, &viewInverse) ;

    // collision detection
    D3DXVECTOR3 v(0.0f, 0.0f, 0.0f);
    if(D3DXSphereBoundProbe(box.minPoint, box.maxPoint &ray._origin, &ray._direction))
    {
        g_pd3dDevice->SetRenderState(D3DRS_FILLMODE, D3DFILL_SOLID);
    }
    break ;
}