I'm not sure about this transformation:
"rgb = mat3( 1, 1, 1, \n"
"0, -0.18732, 1.8556, \n"
"1.57481, -0.46813, 0)*yuv;\n"
In refreshing my memory on matrix * vector operations in GLSL using this page as reference, I think you either need to transpose the coefficient matrix, or move yuv to the front of the operation, i.e., yuv * mat3(...)
. Performing the operation as mat3(...) * yuv
means:
r = y * 1 + u * 1 + v * 1
g = y * 0 + u * -0.18732 + v * 1.8556
b = y * 1.57481 + u * -0.46813 + v * 0
And these conversions are very incorrect.
As another reference, here's a small, complete, sample GLSL shader program that converts YUV -> RGB that may be of some guidance: http://www.fourcc.org/source/YUV420P-OpenGL-GLSLang.c