FFMPEG: chroma key / greenscreen filter for images / video

Question

I need for a video to use chroma key filter / greenscreen filter and overlay it over another video and output it as new video.

Are there any existing libraries, scripts, filters or solutions for this purpose?

Answer 1

The answer (now) is yes, there is a filter for generating chroma-keys and overlaying them. The filter name is "color key". There are examples on the site, here's the command:

ffmpeg -i <base-video> -i <overlay-video> -filter_complex '[1:v]colorkey=0x<color>:<similarity>:<blend>[ckout];[0:v][ckout]overlay[out]' -map '[out]' <output-file>

where <color> is the rgb color to match in hex (ex: 0x000000 for black), <similarity> is a tolerance on the color-match (ex: 0.3), and <blend> (ex: 0.2) controls whether the opacity is on-off or how gradual it is. (See the documentation for more).

Answer 2

Minimal runnable example with test data

The answer at https://stackoverflow.com/a/32291842/895245 was correct, here's just a minimal concrete example of that.

Download input media:

wget https://github.com/cirosantilli/media/raw/master/Ciro_Santilli_with_a_stone_carved_Budai_in_the_Feilai_Feng_caves_near_the_Lingyin_Temple_in_Hangzhou_in_2012.jpg
wget https://github.com/cirosantilli/media/raw/master/opengl-rotating-triangle.mp4

Make the image size match the video size of 1024x1024. The video size can be determined with ffprobe:

convert Ciro_Santilli_with_a_stone_carved_Budai_in_the_Feilai_Feng_caves_near_the_Lingyin_Temple_in_Hangzhou_in_2012.jpg -resize 1024x1024! background.jpg

Do the actual conversion:

ffmpeg -i background.jpg -i opengl-rotating-triangle.mp4 \
  -filter_complex '[1:v]colorkey=0x000000:0.1:[ckout];[0:v][ckout]overlay[out]' \
  -map '[out]' out.mp4

Convert to gif just for previews on this answer:

ffmpeg -i out.mp4 -r 5 -vf "scale=300:-1,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" out.gif

So in my example, I had a black background, which is getting converted into a fixed image.

Outcome preview (horrendous FPS to fit GIF in 2MB for upload here):

Actual video output: https://www.youtube.com/watch?v=3aY6x7u86QQ

Original input files for reference:

opengl-rotating-triangle.mp4

Ciro_Santilli_with_a_stone_carved_Budai_in_the_Feilai_Feng_caves_near_the_Lingyin_Temple_in_Hangzhou_in_2012.jpg

It also just works with a video background.

wget https://upload.wikimedia.org/wikipedia/commons/f/f9/STS-132_Liftoff_Space_Shuttle_Atlantis.ogv
ffmpeg -i STS-132_Liftoff_Space_Shuttle_Atlantis.ogv -i opengl-rotating-triangle.mp4 -filter_complex '[1:v]colorkey=0x000000:0.1:[ckout];[0:v]trim=start=0:end=8[cut0];[cut0][ckout]overlay[out]' -map '[out]' out.mp4

Here I've added another the parameter shortest=1 to the overlay filter:

overlay=shortest=1

to make the overlay stop as soon as the triangle video stops, otherwise it defaults to stopping when the much longer Atlantis video ends.

You might also want to use the trim filter instead to select an arbitrary segment.

The Atlantis video is 1920x1080, and the final output had that size as well. Preview:

Actual video: https://www.youtube.com/watch?v=HI8XniA2Bk8

Tested on Ubuntu 20.10, FFmpeg 4.3.1.

Answer 3

Not specifically that I know of

Opencv contains all the functions you need to read video, convert to RGB, split the color planes, replace pixels base don color, merge frames and write video.

It's good to research with but it's not going to create a plugin directshow filter to do this automatically

Answer 4

The function ChanVeseBinarize in Mathematica may help, see the first "Application" here: http://reference.wolfram.com/mathematica/ref/ChanVeseBinarize.html

For offline processing, you would import in sequence each image, process it, export the overlay. Eventually, you would create the new video from all the overlay images.

Answer 5

This answer to a similiar question on Superuser suggests using MLT.

From MLT website:

MLT is an open source multimedia framework, designed and developed for television broadcasting. It provides a toolkit for broadcasters, video editors, media players, transcoders, web streamers and many more types of applications. The functionality of the system is provided via an assortment of ready to use tools, XML authoring components, and an extensible plug-in based API. The easiest way to try out and learn MLT is by downloading Shotcut