Aliasing issue with SDL + OpenGL masking

Question

I've been trying to make Worms style destructible terrain, and so far it's been going pretty well...

Snapshot1

I have rigged it so that the following image is masked onto the "chocolate" texture.

CircleMask.png

However, as can be seen on Snapshot 1, the "edges" of the CircleMask are still visible (overlapping each other). I'm fairly certain it has something to do with aliasing, as mask image is being stretched before being applied (that, and the SquareMask.png does not have this issue). This is my problem.

My masking code is as follows:

void MaskedSprite::draw(Point pos)
{
    glEnable(GL_BLEND);

    // Our masks should NOT affect the buffers color, only alpha.
    glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_TRUE);

    glBlendFunc(GL_ONE_MINUS_DST_ALPHA,GL_DST_ALPHA);

    // Draw all holes in the texture first.
    for (unsigned i = 0; i < masks.size(); i++)
        if (masks.at(i).mask) masks.at(i).mask->draw(masks.at(i).pos, masks.at(i).size);

    // But our image SHOULD affect the buffers color.
    glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

    // Now draw the actual sprite.
    Sprite::draw(pos);

    glDisable(GL_BLEND);
}

The draw() function draws a quad with the texture on it to the screen. It has no blend functions.

Answer 1

If you invert the alpha channel on your mask image so that the inside of the circle has alpha 0.0, You can use the following blending mode:

glClearColor(0,0,0,1);
// ...
glBlendFunc(GL_DST_ALPHA, GL_ZERO);

This means, when the screen is cleared, each pixel will be set to alpha 1.0. Each time the mask is rendered with blending enabled, it will multiply the mask's alpha value with the current alpha at that pixel, so the alpha value will never increase.

Note that using this technique, any alpha channel in the sprite texture will be ignored. Also, if you are rendering a background before the terrain, you will need to change the blend function before rendering the final sprite image. Something like glBlendFunc(GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA) would work.

Another solution would be to use your blending mode but set the mask texture's interpolation mode to nearest-neighbor to ensure that each value sampled from the mask is either 0.0 or 1.0:

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);

My last bit of advice is this: the hard part about destructible 2D terrain is not getting it to render correctly, it's doing collision detection with it. If you haven't given thought to how you plan to tackle it, you might want to.