Screen Capture in separate thread making Java application slow/unresponsive

Question

I'm working on an application that records the users screen, webcam and microphone whilst he/she is performing certain activities. It will be used for research purposes. The application has been successfully tested on Windows, but on Mac OS X (Maverick with Java 7.0.45) the application becomes slow and unresponsive when recording is started.

This is why I find this difficult to comprehend:

• The recording is done in a separate thread, so how could it influence the responsiveness of another thread? Especially as after each run either Thread.yield() or Thread.sleep(...) are called.
• Logs show that whilst attempting to record at 15 FPS, the resulting frame rate was 2 FPS. So it seems the code that does the capturing of a single frame might be too slow. But why then does it work fine on Windows?

Just a quick note: the application was successfully tested by tons of users on Windows, but I only got to test it on a single Mac. However, that one was just formatted and got a clean install of OS X Maverick, Java (and Netbeans).

Below you will find the code that records the screen and writes it to a video using Xuggler. The code for recording the webcam is similar, and I'd doubt recording the audio has anything to do with it. My question is:

What might be the cause of the application becoming unresponsive?, and

How could the code be made more efficient and so improve FPS?

IMediaWriter writer = ToolFactory.makeWriter(file.getAbsolutePath());
Dimension size = Globals.sessionFrame.getBounds().getSize();
Rectangle screenRect;
BufferedImage capture;
BufferedImage mousePointImg;


writer.addVideoStream(0, 0, ICodec.ID.CODEC_ID_H264, size.width, size.height);

int i = 0;

while (stop == false) {

    // Get mouse cursor to draw over screen image.
    PointerInfo mousePointer = MouseInfo.getPointerInfo();
    Point mousePoint = mousePointer.getLocation();
    Point screenPoint = new Point((int) (mousePoint.getX() - 
        Globals.sessionFrame.getBounds().getX()), (int) (mousePoint.getY() - 
        Globals.sessionFrame.getBounds().getY()));

    // Get the screen image.
    try {
        screenRect = new Rectangle(Globals.sessionFrame.getBounds());
        capture = new Robot().createScreenCapture(screenRect);
    } catch ( ... ) { ... }

    // Convert and resize the screen image.
    BufferedImage image = ConverterFactory.convertToType(capture, 
        BufferedImage.TYPE_3BYTE_BGR);
    IConverter converter = ConverterFactory.createConverter(image, 
        IPixelFormat.Type.YUV420P);

    // Draw the mouse cursor if necessary.
    if (mouseWithinScreen()) {
        Graphics g = image.getGraphics();
        g.drawImage(mousePointImg, (int) screenPoint.getX(), 
            (int) screenPoint.getY(), null);
    }

    // Prepare the frame.
    IVideoPicture frame = converter.toPicture(image, (System.currentTimeMillis() - 
        startTimeMillis()) * 1000);
    frame.setKeyFrame(i % (getDesiredFPS() * getDesiredKeyframeSec()) == 0);

    // Write to the video
    writer.encodeVideo(0, frame);

    // Delay the next capture if we are at the desired FPS.
    try {
        if (atDesiredFPS()) {
            Thread.yield();
        } else {
            Thread.sleep(1000 / getDesiredFPS());
        }
    } catch ( ... ) { ... }

    i++;
}

writer.close();

Answer 1

There are several architectural issues that I can see in your code:

First if you want to execute something at a fixed rate, use the ScheduledThreadPoolExecutor.scheduleAtFixedRate(...) function. It will make your entire delay code part obsolete as well as ensuring that certain OS timing issues will not interfere with your scheduling.

Then to make things faster you need to take your code apart a bit. As far as I can see you have 3 tasks: the capture, the mouse-drawing/conversion and the stream writing. If you put the capture part in a scheduled Runnable, the conversion into multi-parallel execution as Callables into an Executor, and then in a 3rd thread take the results from a result list and write it into the stream, you can fully utilize multi-cores.

Pseudocode:

Global declarations (or hand them over to the various classes):

final static Executor converterExecutor = Executors.newFixedThreadPoolExecutor(Runtime.getRuntime().availableProcessors());
final static LinkedBlockingQueue<Future<IVideoPicture>> imageQueue = new LinkedBlockingQueue<>();
// ...

Capture Runnable (scheduled at fixed rate):

capture = captureScreen();
final Converter converter = new Converter(capture);
final Future<IVideoPicture> conversionResult = converterExecutor.submit(converter);
imageQueue.offer(conversionResult); // returns false if queue is full

Conversion Callable:

class Converter implements Callable<IVideoPicture> {
  // ... variables and constructor

  public IVideoPicture call() {
    return convert(this.image);
  }
}

Writer Runnable:

IVideoPicture frame;
while (this.done == false) {
  frame = imageQueue.get();
  writer.encodeVideo(0, frame);
}

You can ensure that the imageQueue does not overflow with images to render if the CPU is too slow by limiting the size of this queue, see the constructor of LinkedBlockingQueue.