When does VideoToolbox' VTCompressionSession benefit from hardware acceleration?

Question

I've been working on the gstreamer applemedia encoder plugins and improved the VideoToolbox based video encoding. Running a gstreamer pipeline like:

$ gst-launch-1.0 filesrc location=source.avi ! decodebin  ! vtenc_h264 ! h264parse ! qtmux name=mux ! filesink location=sink.mp4

I was expecting to see a very low CPU usage when encoding h264 video using VTCompressionSession on Mac OS systems. However, on the systems I've tested: Mid 2009 Macbook Pro with GeForce 9600M and Mid 2011 Mac mini with Radeon HD 6630M the encoding still consumes between 80% and 130% CPU - which indicates it's not hardware accelerated.

On which hardware configurations, or given which compression parameters (for example for which kVTCompressionPropertyKey_ProfileLevel) does VTCompressionSession use hardware accelerated encoding?

Answer 1

According to http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/sys/applemedia/vtenc.c you are passing in NULL to VTCompressionSessionCreate() for the encoderSpecification parameter. Create an encoder specification dictionary with kVTVideoEncoderSpecification_EnableHardwareAcceleratedVideoEncoder set to kCFBooleanTrue.

Answer 2

The above pipeline will not help you much on determining whether it's actually the encoding process that's responsible for the high CPU usage. There is no sync with the clock in the stream, which means the whole process of decode/encode will go as fast as it can.

Since decodebin is probably using a software decoder, the high CPU usage you are experiencing is most likely due to the decoding process.

I would recommend to compare the output with:

gst-launch-1.0 videotestsrc is-live=true ! vtenc_h264 ! qtmux ! filesink location=test.mp4

Note specially the property "is-live=true" which is instructing videotestsrc to act as a live source and therefore pushing buffers at a constant rate and not as fast as downstream can consume them.