Extract frames from a video in real-time android

Question

I'm doing an app to record a video. What I would like is that when the app is recording the video, each frame is also saved in an arraylist of RGB values in order to extract particular information from it. I know that the two processes (video recording and extraction of frames) are asynchronous, but this is not a problem: the process of extraction can finish after the video recording.

Can someone tell me how I can extract the frames from the video?

Thanks very much.

Answer 1

Using the camera object you can override a function setPreviewCallback. from that function you would get an array of byte data. Which you can convert to bitmap or you can save them in an array of array.

Assuming that you were extends SurfaceView and Implements SurfaceHolder.Callback

Code looks like,

mCamera.setPreviewCallback(new PreviewCallback() {
    public void onPreviewFrame(byte[] data, Camera camera) {

        //do some operations using data

    }
});

If you want to get the RGB value you can use this function,

static public void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width,
            int height) {
        final int frameSize = width * height;

        for (int j = 0, yp = 0; j < height; j++) {
            int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
            for (int i = 0; i < width; i++, yp++) {
                int y = (0xff & ((int) yuv420sp[yp])) - 16;
                if (y < 0)
                    y = 0;
                if ((i & 1) == 0) {
                    v = (0xff & yuv420sp[uvp++]) - 128;
                    u = (0xff & yuv420sp[uvp++]) - 128;
                }

                int y1192 = 1192 * y;
                int r = (y1192 + 1634 * v);
                int g = (y1192 - 833 * v - 400 * u);
                int b = (y1192 + 2066 * u);

                if (r < 0)
                    r = 0;
                else if (r > 262143)
                    r = 262143;
                if (g < 0)
                    g = 0;
                else if (g > 262143)
                    g = 262143;
                if (b < 0)
                    b = 0;
                else if (b > 262143)
                    b = 262143;

                rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000)
                        | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
            }
        }
    }

I think this is what you want as an answer. If i am wrong, please let me know whats is your actual requirement

For getting the green value, you can use the below code,

static public void calculateColor(int[] rgb, int[] color,
            int width, int height, int component) {
        for (int bin = 0; bin < 256; bin++) {
            color[bin] = 0;
        } // bin

        int start = 0;
        int end = width * height;

        if (component == 0) // red
        {
            for (int pix = start; pix < end; pix += 3) {
                int pixVal = (rgb[pix] >> 16) & 0xff;
                color[pixVal]++;
            } // pix
        } else if (component == 1) // green
        {
            for (int pix = start; pix < end; pix += 3) {
                int pixVal = (rgb[pix] >> 8) & 0xff;
                color[pixVal]++;
            } // pix
        } else // blue
        {
            for (int pix = start; pix < end; pix += 3) {
                int pixVal = rgb[pix] & 0xff;
                color[pixVal]++;
            } // pix
        }
    }

Calling this function would return the value of green for every pixel. If you wants the green value for a particular pixel, you have to modify this code.

Answer 2

Sample code for extracting frames from a .mp4 file can be found on the bigflake site. It uses MediaCodec, available on API 16+. If you need to run on older devices, your best bet is probably ffmpeg.

RGB data requires 3 bytes per pixel. 10 seconds of 720p video at 30 fps will require 1280*720*3*30*10 = 791 megabytes. Unless your video frames are fairly small, holding on to a large number of them may not be practical (or possible).

Answer 3

It seems to me you want to get the image while video recording is ongoing.

In that case you will need to look into Build your camera app. This page describes how to record a video using android camera class.

You can then get yuv images through Camera preview callback.