Problems with photoshop-like “lighten” fragment shader

Question

I'm tagging this in OpenGL because I think it's relevant to that space (albeit a different naming).

I'm trying to get photoshop-like blending to work in Flash 11 which uses OpenGL (ES2?). I'm having particular issues with lighten, which I believe should be:

outputColor.rgb = max(base.rgb, blend.rgb);

I believe I am having issues because my layers have transparency. However, even when I pre-multiply my layers, I still am having issues (the image is too dark still).

Am I missing a step here? (This is AGAL, Adobe's gl assembly)

[Shader.Fragment]
tex ft0, v0, fs0 <2d,nearest,nomip,clamp>
tex ft1, v0, fs1 <2d,nearest,nomip,clamp>

// premultiply
mul ft2, ft1.xyzw, ft1.wwww

// premultiplied values
max oc, ft0, ft2

Answer 1

Pay attention, you are also multiplying alpha by itself.

So instead of having ( r.a, g.a, b.a, a ), you are outputting ( r.a, g.a, b.a, a.a ), which will lead to darker image if blended with a black background

You may try doing this :

// premultiply
mul ft2, ft1.xyzw, ft1.wwww
mov ft2.w, ft1.w

// premultiplied values
max oc, ft0, ft2

Answer 2

As crazyjul mentioned, you have to restore original color values from alpha-premultiplied ones. I believe that the following code should work:

tex ft0, v0, fs0 <2d, nearest, nomip, clamp>
tex ft1, v0, fs1 <2d, nearest, nomip, clamp>

max ft0, ft0, fc0
div ft0.xyz, ft0.xyz, ft0.www // obtain original color 1

max ft1, ft1, fc0
div ft0.xyz, ft0.xyz, ft0.www // obtain original color 2

max ft1.xyz, ft0.xyz, ft1.xyz // lighten

mul ft1.xyz, ft1.xyz, ft1.www // pre-multiply output color with the second alpha
mov oc, ft1

Lines max ft0, ft0, fc0 and max ft1, ft1, fc0 are necessary to avoid divizion by zero alpha value. You can set fc0 constant to contain smth like 0,0,0,0.001.

Also note that we use alpha of the second layer (ft1) as the resulting alpha. I don't know whether it is correct.