Displaying camera preview using DirectX Texture2D causes oscillation on Windows Phone 8

Question

I am currently writing a small app which shows the preview from the phone camera using a SharpDX sprite batch. For those who have an nokia developer account, the code is mainly from this article.

Problem

Occasionally, it seems like previous frames are drawn to the screeb (the "video" jumps back and forth), for the fracture of a second, which looks like oscillation/flicker.

I thought of a threading problem (since the PreviewFrameAvailable event handler is called by a different thread than the method which is responsible for rendering), but inserting a lock statement into both methods makes the code too slow (the frame rate drops below 15 frames/sec).

Does anyone have an idea how to resolve this issue or how to accoplish thread synchronization in this case without loosing too much performance?

Code

First, all resources are created, whereas device is a valid instance of GraphicsDevice:

spriteBatch = new SpriteBatch(device);
photoDevice = await PhotoCaptureDevice.OpenAsync(CameraSensorLocation.Back, captureSize);
photoDevice.FocusRegion = null;

width = (int)photoDevice.PreviewResolution.Width;
height = (int)photoDevice.PreviewResolution.Height;

previewData = new int[width * height];

cameraTexture = Texture2D.New(device, width, height, PixelFormat.B8G8R8A8.UNorm);

photoDevice.PreviewFrameAvailable += photoDevice_PreviewFrameAvailable;

Then, whenever the preview frame changes, I set the data to the texture:

void photoDevice_PreviewFrameAvailable(ICameraCaptureDevice sender, object args)
{
    sender.GetPreviewBufferArgb(previewData);
    cameraTexture.SetData(previewData);
}

Finally, the Texture is drawn using a SpriteBatch whereas the parameters backBufferCenter, textureCenter, textureScaling and Math.Pi / 2 are used to center and adjust the texture in landscape orientation:

spriteBatch.Begin();

spriteBatch.Draw(cameraTexture, backBufferCenter, null, Color.White, (float)Math.PI / 2, textureCenter, textureScaling, SpriteEffects.None, 1.0f);

spriteBatch.End();

The render method is called by the SharpDX game class, which basically uses the IDrawingSurfaceBackgroundContentProvider interface, which is called by the DrawingSurfaceBackgroundGrid component of the Windows Phone 8 runtime.

Solution

Additional to Olydis solution (see below), I also had to set Game.IsFixedTimeStep to false, due to a SharpDX bug (see this issue on GitHub for details).

Furthermore, it is not safe to call sender.GetPreviewBufferArgb(previewData) inside the handler for PreviewFrameAvailable, due to cross thread access. See the corresponding thread in the windows phone developer community.

Answer 1

My Guess

As you guessed, I'm also pretty sure this may be due to threading. I suspect that, for example, the relatively lengthy SetData call may be intercepted by the Draw call, leading to unexpected output.

Solution

The following solution does not use synchronization, but instead moves "critical" parts (access to textures) to the same context.

Also, let's allocate two int[] instead of one, which we will use in an alternating fashion.

Code Fragments

void photoDevice_PreviewFrameAvailable(ICameraCaptureDevice sender, object args)
{
    sender.GetPreviewBufferArgb(previewData2);
    // swap buffers
    var previewDataTemp = previewData1;
    previewData1 = previewData2;
    previewData2 = previewDataTemp;
}

Then add this to your Draw call (or equal context):

cameraTexture.SetData(previewData1);

Conclusion

This should practically prevent your problem since only "fully updated" textures are drawn and there is no concurrenct access to them. The use of two int[] reduces the risk of having SetData and GetPreviewBufferArgb access the same array concurrently - however, it does not eliminate the risk (but no idea if concurrent access to the int[] can result in weird behaviour in the first place).