Has anybody used SIKULI for testing their GUI-based apps?

Question

SIKULI seems to have an enormous amount of potential. Has anybody tried to use this as a tool for testing? Or would it be better suited for automating actions for users?

Answer 1

Quoting Unit Testing for GUI (in the project Documentation):

Sikuli is designed to support unit testing for GUI by integrating with junit. The unit testing panel can be opened by clicking on View/Unit Test or by shortcut Cmd-U on Mac (or Ctrl-U on Windows/Linux).

So, while my understanding is that SIKULI is initially aimed at GUI automation, it can definitely be used for GUI testing (which is closely related if you consider that GUI testing = GUI automation + verification framework). Have a look at Unit testing for GUI (JEdit) for a full example (and see the assertXXX on images).

And indeed, I see a big potential in SIKULI for testing as it seems to make writing tests very easy, even without a single line of the real application written (just using some initial mockups for example). SIKULI could become a great companion for various flavor of testing (BDD, acceptance testing, etc).

It' really an amazing piece of software, very impressive.

Answer 2

I am using Sikuli extensively for UI test automation. I am "late" to the Sikuli party, having discovered it in January 2011. I am actually glad I discovered it late, because while it was promising earlier, I don't think until Sikuli x1.0-rc1 (which happened in December) was released that it was ready for primetime.

Previously, I have used TestQuest and EggPlant for UI test automation. In my opinion, Sikuli beats both of them hands down. I truly believe it has the potential to dramatically change how folks perform UI test automation for the better and will be evangelizing it to folks around me.

Using Sikuli correctly is means that you are not following a "record and playback" model. Rather, you have to approach developing test automation with Sikuli - as you need to with all tools - as a software development task.

We are currently in the process of porting a UI automation DSL (Domain Specific Language) we built for EggPlant to Sikuli. One of the key features we will be leveraging in our DSL is Sikuli's text recognition capabilities. This will allow us to run the same script across various localized versions of our product.

Because Sikuli builds on OpenCV (for image recognition) and tesseract-ocr (for text recognition), it has an incredibly amount of power and flexibility.

Answer 3

Also, see the paper: http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-chi2010.pdf

Answer 4

@jordan, Absolutley spot on 'Using Sikuli correctly is means that you are not following a "record and playback" model. Rather, you have to approach developing test automation with Sikuli - as you need to with all tools - as a software development task.

I created an end to end test automation solution for testing a videoconferencing application made by the worlds largest PC manufacturer. They did not understand that it was a full dev project, not a point and click operation that any monkey could execute. Trying to explain the challenges of coding with a dynamically typed language was impossible.

From my experience the biggest challenge is image managment. I used the file system and configparser for the first iteration of the test automation. Using configparser worked however it was difficult to implement. In the future I plan on using blobs. Sikuli does not support direct extraction of images from a DB (yet) although I do have a work around.

Use of an IDE is critical since the Sikuli IDE fails to have and development tools. The 2 IDE's I've configured, NetBeans and Eclipse/PyDev have their own set of issues. The are great for coding however false errors, injection of whitespace and code loss make both less than ideal solutions. I code and test in NetBeans, Execute in SikuliIDE and save all in notepad as a backup.

Inspite of any difficulties encounteres i'm a big supporter of Sikuli. Sikuli has the potential to change test automation, making it accesable to the entire QA community without having to be an OO coder.

Answer 5

Recorded a workflow with a Flex web app. Took a while to figure out a reliable strategy to create the screenshots, but once I did it, the script continued working even after I changed my desktop color scheme! The syntax gets a bit awkward though when you need to click a specific control in a collection of similar controls, i.e. checkboxes, input fields. Looks like the only way to do it is by using find() in combination with right(); left(); inside(). Seems like the smaller the screenshots are, the more reliably are they detected. Imo a good practice would be to include only significant objects on screenshots, and make them as atomic as possible but without compromising their uniqueness.

Answer 6

For less developer centric test automation for Sikuli, also check out RobotFramework.org. There's a tutorial on how to make a (custom) Sikuli test library for Robot Framework

http://blog.mykhailo.com/2011/02/how-to-sikuli-and-robot-framework.html

and I created a simple generic version as well

http://code.google.com/p/simplesikuli

And if there was ever limitations for Sikuli in terms of window handling, GUI controls, mouse and keyboard interaction, you can always complement it with another great free test tool: AutoIt. AutoIt by itself also has limitations, when you combine it with Sikuli, they make up for each tools shortcomings, to replace commercial grade GUI test tools.

Answer 7

Here's my take on the awesomeness of Sikuli for test automation: http://pculture.org/devblogs/mirotesting/2011/06/24/using-sikuli-to-automate-miro-testing/

I have a reliable cross-platform test suite for Miro.

Answer 8

I am actually writing a framework for GUI testing/error handling with sikuli. It's great.

Answer 9

I have used sikuli for GUI testing, also i was able to integrate it with HUDSON.

Answer 10

I have just published my own framework for GUI application testing using Skikuli + RobotFramework.

SikuliFramework provides an object-oriented abstraction on top of Sikuli to assist with interacting GUI elements, such as sets of buttons, checkboxes, radio buttons, windows and dialogue hierarchies for GUI automation and testing. It also has tight integration with RobotFramework.

https://github.com/smysnk/sikuli-framework

Answer 11

Sikuli is based on static image matching. So it is only suitable for situations where the GUI is sufficiently stable. For dynamic GUI such as animation or GUI which includes some kind of randomness, it is not so applicable.

And Sikuli only covers the visual part of the tested. It has no idea if the internal state is indeed as expected.

Answer 12

I am a fan of Sikuli and I believe that it is complementary to other tests and can save much of the manual testing efforts.

However, it takes time to get it right. I gave it a second shot in the span of two years and the second time I was able to achieve good results as I was more familiar with the environment.

It comes with an intuitive IDE and python makes it easy to extend a few things. It's very easy to change click positions, set tolerance and record things. Once you get hold of how to record and check documentation, it becomes easy to use minimum images and improve accuracy. It catches changes in the GUI and results quite easily. It is also easy to wait for a certain event. Error checking is easy.

The biggest issue with it is that the recorded tests often fail if you just run it on another machine than the one it was recorded on. It might have to do with the image comparison based pattern matching. One can improve the match pattern probability by giving tolerance values. But it gets annoying at times to keep on changing the tolerance. My recommendation is to use a different set of images on different platforms and preferably run it on a single machine or VM.

Once you have a common set of workflows, you can make a library of those common functions such as (open-close-save project, change settings)and use across different scripts. It becomes very easy as the library gets comprehensive. It also means that you will have to change the script only at a single location and it will reflect in all the scripts.

I also created a simple framework (image attached) to run the tests and record results using C# .Net. One can make use of anything to create a simple test run application. It just needs to run the test on command line and check the results.

I was working in a small team in which we had limited testing resources. With Sikuli I actually saved efforts of existing QA team and also helped myself catch the bugs before pushing it to main.

I recommended Sikuli to members of other teams in my company and they use it to generate the dataset for the ML models. They did it by automating their Engg application with parameters.

Sikuli takes time to sink in initially. But when done right, it could save lots of efforts.