How can I unit test a GUI?

Question

The calculations in my code are well-tested, but because there is so much GUI code, my overall code coverage is lower than I'd like. Are there any guidelines on unit-testing GUI code? Does it even make sense?

For example, there are graphs in my app. I haven't been able to figure out how to automate the testing of the graphs. It takes a human eye, AFAIK, to check if the graph is correct.

(I'm using Java Swing)

Answer 1

Designs like MVP and MVC typically try to abstract as much logic out of the actual GUI as possible. One very popular article about this is "The Humble Dialog Box" by Michael Feathers. Personally I've had mixed experiences with trying to move logic out of the UI - sometimes it's worked very well, and at other times it's been more trouble than it's worth. It's somewhat outside my area of expertise though.

Answer 2

Of course, the answer is to use MVC and move as much logic out of the GUI as possible.

That being said, I heard from a coworker a long time ago that when SGI was porting OpenGL to new hardware, they had a bunch of unit tests that would draw a set of primatives to the screen then compute an MD5 sum of the frame buffer. This value could then be compared to known good hash values to quickly determine if the API is per pixel accurate.

Answer 3

You can try UISpec4J is an Open Source functional and/or unit testing library for Swing-based Java applications...

Answer 4

Here is some tips:

Try to remove the most code you can from the GUI (have controller, and model object) this way you will be able to test them without the GUI.

For the graphic, you should test the value that you supply to the code that generate the graphic.

Answer 5

You can try to use Cucumber and Swinger for writing functional acceptance tests in plain english for Swing GUI applications. Swinger uses Netbeans' Jemmy library under the hood to drive the app.

Cucumber allows you to write tests like this:

 Scenario: Dialog manipulation
    Given the frame "SwingSet" is visible
    When I click the menu "File/About"
    Then I should see the dialog "About Swing!"
    When I click the button "OK"
    Then I should not see the dialog "About Swing!"

Take a look at this Swinger video demo to see it in action.

Answer 6

There is Selenium RC, which will automate testing a web based UI. It will record actions and replay them. You'll still need to walk through the interactions with your UI, so this will not help with coverage, but it can be used for automated builds.

Answer 7

Testing is an art form. I agree the logic should be removed GUI as much as possible. We can then focus our unit testing there. Like anything else testing is about reducing risk. You dont always need to test everything but alot of times the best thing is to break the different testing up for at different areas.

The other question is what are you really trying to test at the UI layer. UI testing is the most expensive testing because it usually takes longer to create, maintain and it is the most brittle. If you test the logic to know the coordinates are correct prior to trying to draw the line what are you testing specifically? If you want to test a graph with a red line is drawn. Can you give it a set predetermined coordinates and test if certain pixels are red or not red? As suggested above bitmap comparisons work, Selenium but my main focus would be not to over test the GUI but rather test the logic that will help create the UI and then focus on what portion of the UI breaks or is suspect and focus a handful of tests there.

Answer 8

You can use JFCUnit to test your GUI, but graphics can be more challenging. I have on a couple of occasions taken snapshots of my GUI and automatically compared it to a previous version. While this doesn't provide an actual test, it does alert you if an automatic build fails to produce the expected output.

Answer 9

What I gather from your question is that you're looking for an automated way to test your GUI behavior in detail, the example you give is testing whether a curve is actually drawn correctly.

Unit testing frameworks provide a way to do automated testing, but I think the kind of tests you want to do are complicated integration tests that verify the correct behavior of a multitude of classes, among which the classes of your GUI toolkit/library, which you should not want to test.

Your options depend very much on what platforms/toolkits/frameworks you use: for example, an application using Qt as its GUI framework can use Squish to automate its tests. You verify the results of your tests once, and subsequent automatically executed tests compare results against the verified results.

Answer 10

Window Licker for Swing & Ajax

Answer 11

From what I know, this is quite complicated, and really depends on the language - numerous languages have their own way of testing GUI, but if you really need to test the GUI (as opposed to model/gui interaction), you often have to simulate an actual user clicking buttons. For example, the SWT framework used in Eclipse provides SWTBot, JFCUnit has already been mentioned, Mozilla has their own way of simulating this in XUL (and from what I've read on their blogs, these tests seem to be quite brittle).

Sometimes you have to take a screenshot, and test for pixel-perfect rendering (I believe that Mozilla does this to check for correctly rendered pages) - this requires a lengthier setup, but might be what you need for the graphs. That way, when you update your code and a test breaks, you have to manually check the image if the failure was real, or you improved the graph rendering code to generate prettier graphs and need to update the screenshots.

Answer 12

If you are using Swing, FEST-Swing is useful for driving your GUI and testing assertions. It makes it pretty straightforward to test things like "if I click button A, dialog B should be displayed" or "if I select option 2 from the drop-down, all the checkboxes should become deselected".

The graph scenario that you mention is not so easy to test. It's pretty easy to get code coverage for GUI components just by creating and displaying them (and perhaps driving them with FEST). However, making meaningful assertions is the hard part (and code coverage without meaningful assertions is an exercise in self-deception). How do you test that the graph was not drawn upside-down, or too small?

I think that you just have to accept that some aspects of GUIs cannot be tested effectively by automated unit tests and that you will have to test them in other ways.

Answer 13

It's not your job to test the GUI library. So you can dodge the responsibility to check what is actually drawn on screen and check widgets' properties instead, trusting the library that they accurately represent what is drawn.