iOS: How do subtract two video frame with GPU power?

Question

I want to do two RGB video frames subtraction with GPU processing power. i.e., The R,G,B value of each pixels of the first image subtract with the second image. Also, the result want to have negative RGB value and store in NSMutableArray. I use AVCaptureSession to help me get the raw video frame in CMSampleBufferRef format.

1) Any image format support negative RGB value? Or I need to make a matrix-like class to store the image?

(Seems like some image format use float to store the RGB value...)

(I have heard of Image Different Blend Mode, it is similar to what I want but will absolute the RGB value, I want the result to have negative value)

2) How can I do the subtraction with GPU processing power?

(I can do it with CPU, but the FPS is too slow, around 5-6 FPS, but I want it to be 30 FPS)

(Are OpenGL or Quartz Composer helpful?)

Thanks.

Answer 1

As Hammer points out, the GPUImageDifferenceBlendFilter in my open source GPUImage framework does just this. It uses the following fragment shader to obtain the absolute difference of the red, green, and blue channels:

 varying highp vec2 textureCoordinate;
 varying highp vec2 textureCoordinate2;

 uniform sampler2D inputImageTexture;
 uniform sampler2D inputImageTexture2;

 void main()
 {
     mediump vec4 textureColor = texture2D(inputImageTexture, textureCoordinate);
     mediump vec4 textureColor2 = texture2D(inputImageTexture2, textureCoordinate2);
     gl_FragColor = vec4(abs(textureColor2.rgb - textureColor.rgb), textureColor.a);
 }

If you need to compare consecutive video frames, you can also use a GPUImageBuffer to delay an older frame to be processed along with the current video frame. I have an example of this in the FilterShowcase sample application within the framework.

I'm not quite sure what you mean by "the result want to have negative RGB value." The only texture format supported on current iOS devices for texture-backed framebuffers (your render targets) is 32-bit BGRA. This means that each color channel ranges from 0-255, with no negative values allowed. You could use the bottom half of that color range for negative values, and the top for positive, though.

You could modify my GPUImageSubtractBlendFilter to do that, using a fragment shader like the following:

 varying highp vec2 textureCoordinate;
 varying highp vec2 textureCoordinate2;

 uniform sampler2D inputImageTexture;
 uniform sampler2D inputImageTexture2;

 void main()
 {
     lowp vec4 textureColor = texture2D(inputImageTexture, textureCoordinate);
     lowp vec4 textureColor2 = texture2D(inputImageTexture2, textureCoordinate2);

     gl_FragColor = vec4((textureColor.rgb - textureColor2.rgb + 1.0) * 0.5, textureColor.a);
 }

You'd then need to convert the 0-255 output to -128 - 128 (or whatever your range is) after you read the pixels back out.