Choppy frame rate using a SourceDataLine

Question

I've been programming sound with simple Swing graphics for a little while, but my frame rates are choppy for some reason.

Generally I'm doing something like the following on a background thread:

for(;;) {
    // do some drawing
    aPanel.updateABufferedImage();
    // ask for asynchronous repaint
    aPanel.repaint();

    // write the sound
    aSourceDataLine.write(bytes, 0, bytes.length);
}

Through debugging, I think I've already traced the problem to the blocking behavior of SourceDataLine#write. Its doc states the following:

If the caller attempts to write more data than can currently be written [...], this method blocks until the requested amount of data has been written.

So, what this seems to mean is SourceDataLine actually has its own buffer that it is filling when we pass our buffer to write. It only blocks when its own buffer is full. This seems to be the holdup: getting it to block predictably.

To demonstrate the issue, here's a minimal example which:

• writes 0's to a SourceDataLine (no audible sound) and times it.
• draws an arbitrary graphic (flips each pixel color) and times the repaint cycle.

import javax.swing.*;
import java.awt.*;
import java.awt.event.*;
import java.awt.image.*;
import javax.sound.sampled.*;

class FrameRateWithSound implements Runnable {
    public static void main(String[] args) {
        SwingUtilities.invokeLater(new FrameRateWithSound());
    }

    volatile boolean soundOn = true;
    PaintPanel panel;

    @Override
    public void run() {
        JFrame frame = new JFrame();
        JPanel content = new JPanel(new BorderLayout());

        final JCheckBox soundCheck = new JCheckBox("Sound", soundOn);
        soundCheck.addActionListener(new ActionListener() {
            @Override
            public void actionPerformed(ActionEvent e) {
                soundOn = soundCheck.isSelected();
            }
        });

        panel = new PaintPanel();

        content.add(soundCheck, BorderLayout.NORTH);
        content.add(panel, BorderLayout.CENTER);

        frame.setContentPane(content);
        frame.pack();
        frame.setLocationRelativeTo(null);
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.setVisible(true);

        new Thread(new Worker()).start();
    }

    class Worker implements Runnable {
        @Override
        public void run() {
            AudioFormat fmt = new AudioFormat(
                AudioFormat.Encoding.PCM_SIGNED,
                44100f, 8, 1, 1, 44100f, true
            );

            // just 0's
            byte[] buffer = new byte[1000];

            SourceDataLine line = null;
            try {
                line = AudioSystem.getSourceDataLine(fmt);
                line.open(fmt);
                line.start();

                for(;;) {
                    panel.drawNextPixel();
                    panel.repaint();

                    if(soundOn) {
                        // time the write
                        long t = System.currentTimeMillis();

                        line.write(buffer, 0, buffer.length);

                        t = ( System.currentTimeMillis() - t );
                        System.out.println("sound:\t" + t);
                    }

                    // just so it doesn't fly off the handle
                    Thread.sleep(2);
                }
            } catch(Exception e) {
                // lazy...
                throw new RuntimeException(e);
            } finally {
                if(line != null) {
                    line.close();
                }
            }
        }
    }

    class PaintPanel extends JPanel {
        Dimension size = new Dimension(200, 100);

        BufferedImage img = new BufferedImage(
            size.width, size.height, BufferedImage.TYPE_INT_RGB);

        int x, y;

        int repaints;
        long begin, prev;
        String fps = "0";

        PaintPanel() {
            setPreferredSize(size);
            setOpaque(false);

            Graphics2D g = img.createGraphics();
            g.setColor(Color.LIGHT_GRAY);
            g.fillRect(0, 0, size.width, size.height);
            g.dispose();
        }

        synchronized void drawNextPixel() {
            img.setRGB(x, y, img.getRGB(x, y) ^ 0xFFFFFF); // flip

            if( ( ++x ) == size.width ) {
                x = 0;
                if( ( ++y ) == size.height ) {
                    y = 0;
                }
            }
        }

        @Override
        protected synchronized void paintComponent(Graphics g) {
            super.paintComponent(g);
            g.drawImage(img, 0, 0, size.width, size.height, null);

            long curr = System.currentTimeMillis();
            // time this cycle
            long cycle = ( curr - prev );
            System.out.println("paint:\t" + cycle);

            ++repaints;
            // time FPS every 1 second
            if(curr - begin >= 1000) {
                begin = curr;
                fps = String.valueOf(repaints);
                repaints = 0;
            }

            prev = curr;

            g.setColor(Color.RED);
            g.drawString(fps, 12, size.height - 12);
        }
    }
}

I recommend actually running the example if you are curious about this.

A typical System.out feed during "playback" is something like the following:

sound:  0
paint:  2
sound:  0
paint:  2
sound:  0
paint:  3
sound:  0
paint:  2
paint:  2
sound:  325 // <- 'write' seems to be blocking here
sound:  0
paint:  328
sound:  0
paint:  2

This shows the behavior of write pretty clearly: it spins a majority of the time, then blocks for an extended period, at which point the repaints chug as well. The FPS meter typically displays ~45 during playback, but the animation is obviously choppy.

When sound is turned off, FPS climbs and the animation is smooth.

So is there a way to fix it? What am I doing wrong? How can I get write to block at a regular interval?

This behavior is apparent both on Windows and OSX environments.

One thing I've tried is using Thread.sleep to regulate it, but it's not very good. It's still choppy.

Answer 1

The solution seems to be to use open(AudioFormat, int) to open the line with a specified buffer size.

line.open(fmt, buffer.length);

Timing it again, we can see that write blocks much more consistently:

sound:  22
paint:  24
sound:  21
paint:  24
sound:  20
paint:  22
sound:  21
paint:  23
sound:  20
paint:  23

And the animation is smooth.

Answer 2

I seriously doubt that the sound playback is the culprit here. Please see my comment on the main question. The blocking that occurs in the audio write() method pertains to the rate at which the audio is presented to the playback system. Since audio processing is usually an order of magnitude faster than the audio system can play back (limited to 44100 fps), a majority of the time is spent blocking, for BOTH SourceDataLine and Clip. During this form of blocking, the CPU is FREE to do other things. It does not hang.

I'm more suspicious of the your use of synchronization for the images, and the editing being done to the image. I'm pretty sure editing on a bit level will wipe out the default graphics acceleration for that image.

You might check out this link on Graphics2D optimizations at Java-Gaming.org http://www.java-gaming.org/topics/java2d-clearing-up-the-air/27506/msg/0/view/topicseen.html#new

I found it very helpful for optimizing my 2D graphics.

I'm not sure why you are getting coalescence in your particular case. The few times it has been an issue for me is when the looping code for the frame and the component were in the same class. By just putting the "game loop" code and the component in separate classes, the problem always went away for me, so I never bothered to think about it further. As a consequence I don't have a clear understanding of why that worked, or if that action was even a factor.

[EDIT: just looked more closely at your audio code. I think there is room for optimization. There are calculations being redone needlessly and could be consuming cpu. For example, since you have final values in your inner loop, why recalculate that part every iteration? Take the constant part and calculate that into a value once, and only calculate the unknowns in the inner loop. I recommend refactoring to allow avoiding all synchronizing and optimizing the data generation of the audio data, and then seeing if there is still a problem.]