Access raw YUV values from JPEG with JavaScript

Question

I have dynamically loaded a JPEG image into a 2D html canvas and I would like to access the raw YCbCr pixel values.

From what I understand, JPEG encodes the pixel data in YCbCr, but chrome appears convert it to RGB when accessing it with getImageData().

I have tried using the RGB to YCbCr conversion calculations but it appears to be a lossy conversion as they don't map perfectly 1-to-1.

Is it possible to access the raw YCbCr pixel values in JavaScript?

Answer 1

Yes and no.

When the browser loads an image with the native methods (ie. Image element) the YCbCr (in case of JPEGs) are automatically converted to RGB space by the browser and ICC and gamma correction is applied by browsers which support that.

This mean that when the image is loaded and ready for use, it is already in RGB space (which is correct as the monitor is RGB). The canvas does not have a part in this process and will always contain pixels as RGBA. All browsers supporting canvas will therefor return the data as RGBA when using getImageData().

The only way to access raw YCbCr is to do a low-level parse of the raw file yourselves. You can do this in any language that can iterate bytes including JavaScript. For JS I would recommend using typed arrays and DataViews to parse over the data but be warned: this is tedious and prone to errors (as it is a whole project on its own) but fully possible to do if you absolutely need the raw data.

Another way is to convert RGB back to YCbCr but chroma and the aforementioned color- and gamma-corrections will probably affect the result not making it identical to the original raw data (which in any ways is in compressed form). You can also export canvas as JPEG using the toDataURL() method on the canvas element.

Answer 2

I know this is a very old question, but wanted to provide the update that Chrome now rasterizes some JPEG and WebP images directly from YUV planes. You must have GPU rasterization enabled (and Metal disabled) and might sometimes need to enable the feature from chrome://flags

You can run a trace, decode an image, and include the blink category and see if any events pop up for YUV decoding. If so, then you can experiment with trying to directly access the YUV planes from JavaScript, although I'm admittedly doubtful that will work out of the box (for plumbing/security reasons).