First, a quick warning: starting with VLC2.2 (current git version, to be released soon), the size parameter is a size_t. There's no API for smem (yet? hopefully this will change), which sucks, so this would silently break your application.
Then, a quick comment about the "data" parameter: it's supposed to hold what you need to do your processing. That being a pointer to a struct, an instance of a class, you name it. I strongly doubt passing a long long would work on a 32bits machine, since you'd be forcing 64 bits in something which can only contain 32. What you should do is to declare a struct, and store what you need into it. Here, a good example could be:
struct MyParamStruct
{
YourMutexType imageMutex; // Here mutex is not a global variable anymore
int otherParam; // You can use this to store the value 200 that you were passing before
};
//...
// Init the struct somewhere
MyParamStruct* param = new MyStructParam;
param->otherParam = 200;
//...
sprintf(smem_options
, "#transcode{vcodec=h264}:smem{"
"video-prerender-callback=%lld,"
"video-postrender-callback=%lld,"
"video-data=%lld,"
"no-time-sync},"
, (long long int)(intptr_t)(void*)&cbVideoPrerender
, (long long int)(intptr_t)(void*)&cbVideoPostrender //This would normally be useful data, 100 is just test data
, (long long int)(intptr_t)(void*)param
);
About the mutex usage, it looks good to me. Actually it seems that you don't have any concurrency issue here, as you synchronously allocate a new buffer for each frame. If you were using a preallocated buffer each time, you would need to consider locking when exiting the postrender function.
In fact I'm not even sure about what is exactly the void pointer p_video_data.
That depends on your image format. For H264, it would depend on the pixel format that will be output by the decoder. Since you're asking for H264 output, it's quite likely you'll get a planar pixel format, though the exact type would depend on your H264 profile.
If you're expecting rawdata as a result (which seems to be the case, as CV_8UC3 seems to be refering to a 3 channels raw image, after a quick glance at google), I'd recommend you switch to RV32:
#transcode{vcodec=RV32}
What you need to pass to the transcode module is your output fourcc, VLC will deal with the input for you :)
Update
I have no idea if the Mat class takes the ownership of your pointer, but you might want to check that as well.
Update 2
To answer your further question about what is RV32:
/* 24 bits RGB */
#define VLC_CODEC_RGB24 VLC_FOURCC('R','V','2','4')
/* 24 bits RGB padded to 32 bits */
#define VLC_CODEC_RGB32 VLC_FOURCC('R','V','3','2')
/* 32 bits RGBA */
#define VLC_CODEC_RGBA VLC_FOURCC('R','G','B','A')
If you expect 3 bytes only, then you probably should RV24 a try! I should probably have suggested that from the beginning, since the 8CU3 definitely suggests 3 bytes only...