Stencil buffer test settings DirectX9

Question

I have a function that renders a triangle of desired color. I am trying to render a yellow triangle and then a red triangle over it with stencil test enable. I am using a circle as my stencil window. What should be my stencil test equations and operations to get the below output. All rendering in DirectX09 only. Desired output Kindly guide to few good and simple examples for below APIs....

SetRenderState(D3DRS_STENCILFUNC,
SetRenderState(D3DRS_STENCILREF,
SetRenderState(D3DRS_STENCILMASK,
SetRenderState(D3DRS_STENCILWRITEMASK,
SetRenderState(D3DRS_STENCILZFAIL,
SetRenderState(D3DRS_STENCILFAIL,
SetRenderState(D3DRS_STENCILPASS,

How do we use Stencil operation in DirectX09 shaders effect file (vs_3_0 and ps_3_0) ?

Answer 1

The documentation of the renderstates should answer most of your related questions.

For creating the stencilmask, you need the methods

SetRenderState(D3DRS_STENCILZFAIL,D3DSTENCILOP_KEEP)
SetRenderState(D3DRS_STENCILFAIL,D3DSTENCILOP_INCRSAT)
SetRenderState(D3DRS_STENCILPASS,D3DSTENCILOP_INCRSAT)
SetRenderState(D3DRS_STENCILFUNC,D3DCMP_ALWAYS)

because they increment the stencilbuffer, while rendering your circle. Then youre drawing your yellow triangle without using the stencilbuffer. After that youre drawing the red triangle with

SetRenderState(D3DRS_STENCILZFAIL,D3DSTENCILOP_KEEP)
SetRenderState(D3DRS_STENCILFAIL,D3DSTENCILOP_KEEP)
SetRenderState(D3DRS_STENCILPASS,D3DSTENCILOP_KEEP)
SetRenderState(D3DRS_STENCILFUNC,D3DCMP_LESS)    
SetRenderState(D3DRS_STENCILREF,0)

so your that your stenciltest returns only true, where youre circle had been drawn before (there should be the stencilvalue greater 0). If after that there is nothing drawn properly, you should try to deactivate the Z-Test maybe, the order of your triangles isn't right.

How do we use Stencil operation in DirectX09 shaders effect file (vs_3_0 and ps_3_0) ?

Stencil operations are only used from your main program code. Shaders cannot have any effect on the stenciltests.