How to handle alpha in a manual “Overlay” blend operation?

Question

I'm playing with some manual (walk-the-pixels) image processing, and I'm recreating the standard "overlay" blend. I'm looking at the "Photoshop math" macros here:

http://www.nathanm.com/photoshop-blending-math/ (See also here for more readable version of Overlay)

Both source images are in fairly standard RGBA (8 bits each) format, as is the destination. When both images are fully opaque (alpha is 1.0), the result is blended correctly as expected:

But if my "blend" layer (the top image) has transparency in it, I'm a little flummoxed as to how to factor that alpha into the blending equation correctly. I expect it to work such that transparent pixels in the blend layer have no effect on the result, opaque pixels in the blend layer do the overlay blend as normal, and semitransparent blend layer pixels have some scaled effect on the result.

Can someone explain to me the blend equations or the concept behind doing this?

Bonus points if you can help me do it such that the resulting image has correctly premultiplied alpha (which only comes into play for pixels that are not opaque in both layers, I think.)

Thanks!

// factor in blendLayerA, (1-blendLayerA) somehow?
resultR = ChannelBlend_Overlay(baseLayerR, blendLayerR); 
resultG = ChannelBlend_Overlay(baseLayerG, blendLayerG);
resultB = ChannelBlend_Overlay(baseLayerB, blendLayerB);
resultA = 1.0; // also, what should this be??

Answer 1

Just a guess, but I would try

resultA = 1 - (1-baseAlpha) * (1-blendAlpha)

Answer 2

After blending the base color and the blend color, mix the original base color and the color resulting from the blending using the alpha of the blend color:

vec4 baseColor = ...;
vec4 blendColor = ...;
vec4 blendedColor = blend(baseColor, blendColor);
vec4 fragmentColor = (1.0 - blendColor.a) * baseColor + blendColor.a * blendedColor;

I use this for "overlay" blending an opaque base color and a blend texture which has a lot of (semi) transparent pixels.

Answer 3

I was experimenting with this issue exactly, until I found out that the best is to have the base and the blend layer both with straight alpha, then premultiply only the result with the base alpha.