ChromaKeying a video programatically using FLARToolkit

Question

So this will be a bit of a doosy, maybe... I am trying to play a video clip in an Augmented Reality using the Flash plugin called FLARToolkit. I have a green screen video and i need to play the video on a plane. I am currently using a MovieClipMaterial to lay the video on the display plane. I want to itterate through the pixels in the display plane and change the pixels that match the green in the green screen and set those pixels to transparent, chroma keying. There may be a more efficient way of doing this and I am open to suggestions.

I have also looked into ColorMatrixFilters but I havent been able to devise a way to make it selective, rather than filtering out all the green in the clip.

Here is my code as it stands now:

[SWF(width='640', height='480', backgroundColor='#000000', frameRate='40')]

public class HelloFLAR extends Sprite
{
    private var fm:FLARManager;
    private var scene:Scene3D;
    private var view:Viewport3D;
    private var camera:FLARCamera3D;
    private var bre:BasicRenderEngine;
    private var lre:LazyRenderEngine;
    private var p:Plane;
    private var con:DisplayObject3D;
    private var marker:FLARMarker;
    private var v:Vid;
    private var mat:MovieMaterial
    private var bmd:BitmapData;
    private var videoStream:NetStream;
    private var container:FLARBaseNode;
    private var RGBRaster:FLARRgbRaster_BitmapData;

    public function HelloFLAR()
    {
        initFLAR();
        v = new Vid();
        //source for displayed video
        v.vid.source = "Reuters.m4v";
        v.vid.stop();

    }

    private function initFLAR():void
    {
        fm = new FLARManager("flarConfig.xml");
        fm.addEventListener(FLARMarkerEvent.MARKER_ADDED, onAdded);
        fm.addEventListener(FLARMarkerEvent.MARKER_REMOVED, onRemoved);
        fm.addEventListener(Event.INIT, init3D);
        addChild(Sprite(fm.flarSource));


    }

    private function onAdded(e:FLARMarkerEvent):void
    {
        marker = e.marker;
        p.visible = true;
        v.vid.play();
    }

    private function onRemoved(e:FLARMarkerEvent):void
    {
        marker = null;
        p.visible = false;
        v.vid.stop();
    }

    private function init3D(e:Event):void
    {
        scene = new Scene3D();
        camera = new FLARCamera3D(fm.cameraParams);
        camera.z = -30;
        view = new Viewport3D(640, 480, true);
        lre = new LazyRenderEngine(scene,camera,view);


        //Display plane and Videp material
        mat = new MovieMaterial(v, false, true);
        p = new Plane(mat, 320, 240, 2, 2);
        mat.doubleSided = true;
        p.rotationZ = 90;
        p.visible = true;


        //Green Leech from video plane (Attempt at chromakeying)
        var matrix:Array = new Array();         
        matrix = matrix.concat([1,0,0,0,0]); //Red
        matrix = matrix.concat([0,1,0,0,0]); //Green
        matrix = matrix.concat([0,0,1,0,0]); //Blue
        matrix = matrix.concat([0,-1,0,0,0]); //Alpha

        var color_filter:ColorMatrixFilter = new ColorMatrixFilter(matrix);;
        v.filters = [color_filter];

        //container for display Plane
        con = new DisplayObject3D();
        con.addChild(p);


        //Add plane container to scene
        scene.addChild(con);
        addChild(view);
        addChild(new FramerateDisplay());

        addEventListener(Event.ENTER_FRAME, loop);
        //addEventListener(Event.ENTER_FRAME, pixelCheck);

    }

    private function loop(e:Event):void
    {
        if(marker != null)
        {
            con.transform = FLARPVGeomUtils.convertFLARMatrixToPVMatrix(marker.transformMatrix);
        }
        lre.render();
    }

I can't seem to get access to the pixels of the video. I have tried accessing the pixels through the plane and the MovieClipMaterial with no luck.

Any help would be greatly appreciated!

Adeptatus

Answer 1

Yes, you can do that;

1. create transparent background video to create the transparent background video, using green screen background footage, remove the green background on Adobe Afters Effects or Premiere or Final Cut or.. in export or render settings change to RGB + Alpha, and then export out

2. in your code;

   mat = new MovieMaterial(v, false, true);
   mat = new MovieMaterial(v, true, true);
   

Answer 2

Sorry, I forgot, the transparent video format should be flv. Because a few format can carry transparency infomation. It can be tested using River FLV player.