Actionscript 3 + AGAL to fade edges of plane transparent

Question

Im showing 2D content on simple Stage3D plane made of 2 triangles. Content is drawn on BitmapData and uploaded as texture in case of redraw. I also have the AGAL program to give transparency for the whole container.

What I would like to do is to use GPU to give "fade to transparent" effect for the edges of container to blend nicely with another containers. Also to use program constants to control the faded edges.

Currently using this shader code:

vertex assembler:

"m44 op, va0, vc0 \n" + 
"mov v0, va0 \n" + // tell fragment shader about XYZ
"mov v1, va1 \n" + // tell fragment shader about UV
"mov v2, va2\n";   // tell fragment shader about RGBA

fragment assembler:

"tex ft0, v1, fs0 <2d,clamp,linear> \n" + 
"mul ft0.a, ft0.a, fc0.x\n" + // manage alpha value that is set as program constant
"mov oc, ft0 \n"; // move this value to the output color

Any help or hints is really appreciated

Answer 1

Transparency is pretty straight-forward, especially in the simple example you have provided. In this case, what you want to do is use the UV coordinates to create an alpha mask. Since UV vary from 0-to-1, an easy way to do this is, in your fragment program:

//assume [0.5, 2.0, 1.0, 10.0] is in fragment constant 0
sub ft1.xy, va1.xy, fc0.xx  //subtract 0.5 from UV, and store in temporary register 1
mul ft1.xy, ft1.xy, fc0.yy  //multiply UV by 2.0, UV now scaled from -1-to-0-to-1
abs ft1.xy, ft1.xy  //UV now scaled from 1-to-0-to-1
sub ft1.xy, fc0.zz, ft1.xy  //UV now scaled from 0-to-1-to-0
pow ft1.xy, ft1.xy, fc0.ww  //UV has been turned into alpha, with fade controlled by fc0.w
mul ft1.a, ft1.x, ft1.y  //combine U and V alpha into a single alpha value
mul ft0.a, ft0.a, ft1.a  //combine UV-based alpha with existing alpha

Also, don't forget to enable transparency on your context3d with something like this:

if(transparent)
    context3d.setBlendFactors(Context3DBlendFactor.SOURCE_ALPHA, Context3DBlendFactor.ONE_MINUS_SOURCE_ALPHA);
else
    context3d.setBlendFactors(Context3DBlendFactor.ONE, Context3DBlendFactor.ZERO);

Otherwise your transparent pixels will not properly blend.

One last caveat. If you plan on overlapping the transparent bitmap onto any other object in your scene, you absolutely must render objects in your scene in inverse-z order from the camera. Transparency is handled by rendering pixels on top of pixels already rendered in the buffer. If you render a close, transparent object first, then try to render an object behind it, the z-buffer check will prevent the farther object from being drawn, and you will not get the transparent look you're after.