Porting OpenGL stencil functionality to DirectX 11

Question

I have some rendering code written in OpenGL. I use stencil buffer to implement clipping:

    //Let's assume this is done in render loop.

    if(!already_created())
    {
      create_stencil_attachment_and_bind_to_FB_as_depth_stencil_attachment();
    }

    glEnable(GL_STENCIL_TEST);
    glColorMask(0,0,0,0);
    glDepthMask(0);
    glClearStencil(0);
    glClear(GL_STENCIL_BUFFER_BIT);
    glStencilFunc(GL_ALWAYS,1,1);
    glStencilOp(GL_REPLACE,GL_REPLACE,GL_REPLACE);

    render_to_stencil();

    glColorMask(1,1,1,1);
    glDepthMask(1);
    glStencilFunc(GL_EQUAL,1,1);
    glStencilOp(GL_KEEP,GL_KEEP,GL_KEEP);

    render_with_clipping();

    glDisable(GL_STENCIL_TEST);

Now, the problem is, I need to port this code to DX11. I saw examples on MSDN and some nice tutorials. I end up with this logic:

1. Create ID3D11Texture2D with format = DXGI_FORMAT_D32_FLOAT_S8X24_UINT.
2. Create ID3D11DepthStencilState for rendering to stencil: //Let's call it DS_RENDER
   - For both front and back faces:
     - op = D3D11_STENCIL_OP_REPLACE for all 3 cases
     - func = D3D11_COMPARISON_ALWAYS
   - DepthEnable = FALSE
   - StencilEnable = TRUE
   - StencilReadMask = 0xFF
   - StencilWriteMask = 0xFF
3. Create ID3D11DepthStencilView for state and texture created before. //Let's call it DSV
4. Create ID3D11DepthStencilState for using stencil as 'input': //Let's call it DS_CLIP
   - For both front and back faces:
     - op = D3D11_STENCIL_OP_KEEP for all 3 cases
     - func = D3D11_COMPARISON_EQUAL
   - DepthEnable = FALSE
   - StencilEnable = TRUE
   - StencilReadMask = 0xFF
   - StencilWriteMask = 0xFF

Now, I'm not sure how do I set stencil as target or input.

MSDN says:

`pDevice->OMSetDepthStencilState(pDSState, 1);`

and

`pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, pDSV);`

If I understand these calls correctly, first one sets stencil state, while the second one binds pDSV as additional 'attachment' to render target. Is that correct?

If so, will this work as I expect?

    pDevice->OMSetDepthStencilState(DS_RENDER, 1);
    pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, DSV);

    render_geometry_to_stencil_buffer();

    pDevice->OMSetDepthStencilState(DS_CLIP, 1);

    render_geometry_with_clipping();

    pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, NULL); //Does this disable stencil testing?

Thanks in advance for every help or useful hint.

Answer 1

If you want to render only to stencil, use (setting your state for writing):

pd3dDeviceContext->OMSetRenderTargets(0, NULL, DSV);

You don't need to render to color buffer, so no need to bind it.

Then to render to your target and enabling stencil test, use:

pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, DSV);

When you use stencil as input, a very simple thing is also to set StencilWriteMask = 0; So it will never write to it (which is what you want to render clipped geometry).

If you use:

pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, NULL); 

You will indeed disable any form of depth/stencil test (no more depth bound, so DepthStencilState will have no effect at all).

Also I would use DXGI_FORMAT_D24_UNORM_S8_UINT for you depth format (personal preference tho), it will prefectly fit your use case and consume less memory.

Hope that helps.