TestingGUIApplications

By "GUI Applications" I mean ones where Java draws the GUI (e.g. with Swing or SWT or other technologies), not ones where the application is a servlet or JSP or similar concept running inside a web server. For discussion of testing those applications, see TestingWebApplications. In the java-gui-testing group (http://groups.yahoo.com/group/java-gui-testing/) we regularly get off-topic questions concerning automating tests of Web Applications, so I wanted to clarify the purpose of this page up front.

Test automation tools

A well tested application is going to have greater chance of success. GUI applications are expensive to test because their nature involves human interaction which tends to require manual testing (high expense). With a given budget, the hope is that you can achieve more testing by automating your manual tests. It is my (David Herron) belief that testing, and the existance of automated testing tools for Windows, is an overlooked reason why Windows is so successful in the marketplace. That is, because the Windows platform has a wide choice of GUI test automation tools, this means applications developers for Windows can afford to do more testing, and as a result it's likely they'll find more bugs in their application. Fewer GUI automation tools support other platforms (e.g. Mac OS, Linux, Solaris, other Unixen); one exception is Rational Functional Tester for Linux. Fortunatelly the Java platform has some GUI automation tools, as we see listed here:

Known tools:

• Abbot: http://abbot.sourceforge.net supports both record/playback and programmatic automation of Swing, programmatic automation of SWT
• Eggplant: VNC based GUI testing http://www.redstonesoftware.com/
• FEST: http://code.google.com/p/fest/ Provides an easy-to-use API for creating Swing GUI tests using JUnit and/or TestNG. Features include screenshots of failing tests embedded in HTML test reports. It will soon support a Groovy builder.
• froglogic Squish/Java: Advanced automated Java GUI testing tool with tight Swing/AWT and SWT/RCP/Eclipse integration and support http://www.froglogic.com
• GUIdancer: http://www.bredex.de/en/guidancer/ tests can be specified prior to implementation; no programming know-how needed; commercial
• IBM Rational Functional Tester: http://www.ibm.com/software/awdtools/tester/functional/ (commercial) supports web and .Net as well as Swing and SWT GUIs
• JFCUnit: JUnit extension for testing Java GUIs http://jfcunit.sourceforge.net
• Jacareto: Capture and playback for Java GUIs http://www.ph-ludwigsburg.de/mathematik/personal/spannagel/jacareto/
• jDiffChaser: Java GUIs capture, playback and screen (visual) comparisons between last production and development. Can be remotely used and automated. http://jdiffchaser.sourceforge.net
• Jemmy: Jemmy GUI Testing Tool http://jemmy.netbeans.org
• Marathon: a gui testing tool http://marathonman.sourceforge.net
• Pounder: Java GUI tester with record/playback http://pounder.sourceforge.net
• qftest: http://www.qfs.de/en/qftestJUI/index.html (commercial)
• UISpec4J: http://www.uispec4j.org/, an elegant API for implementing functional and/or unit tests for Swing-based applications on top of JUnit
• VNCPlay: Interactive Session Replay and Performance Measurement http://suif.stanford.edu/vncplay/
• WindowTester: Record JUnit tests for Java GUIs... all Java... no XML (commercial) http://downloads.instantiations.com/WindowTesterDoc/integration/latest/docs/html/toc.html
• XEUS: http://www.xeus-tool.com/ (commercial), records the test script in XML, and bases all its discovery on image matching

Most of these build on the java.awt.Robot class. The Robot class exists to provide the minimal core features required to support any test automation tool. Robot synthesizes native-level events so that when they arrive in the application they appear to be indistinguishable from events created by users playing with their mouse and keyboard. In some cases the automation tools instead send AWT events through the AWT event queue, bypassing the native event layer. There is an important distinction between these approaches in the amount of the Java platform exercised during the test. Obviously a native level event (as produced by Robot) will go through the full AWT "stack" while simply pushing an Event object into the queue does not. Deep in the bowels of AWT there is obviously code that recieves OS-specific events, translating them into AWT event(s). Depending on your testing goals the push AWT events into the queue model may be enough for you, just know that your test will not be "complete" in that it will not cover the whole AWT "stack".

Record/Playback vs. Programmatic

These are the two general styles of GUI automation tools.

• Record/playback: The user records a set of actions on the GUI Under Test, and the tool is able to replay those actions later.
• Programmatic: The user writes code of some kind describing the interaction with the GUI Under Test.

This is a long-standing dispute in the testing industry over which method is more appropriate. So it's unlikely we can provide an answer on this web page, so instead let's try and clearly state the pro's and con's of each approach.

Record/Playback

• Pros
  • No explicit programming required to get started
  • Can quickly generate the framework for a long test
  • Avoids having to manually generate most of your test components
• Cons
  • People assume once you've recorded an action, testing is done
  • Sometimes fragile, in that tests break if the GUI changes
  • Generated tests usually require some manual intervention to focus on what is supposed to be tested
  • How do you properly determine delay factors between events being synthesized?
  • Often have to re-record tests

Programmatic

• Pros
  • Can adjust to changes in GUI
  • Can be included in refactoring
  • Can determine when to send the next event more correctly than simple sleep's
  • The test developer has tons of flexibility available
• Cons
  • The test developer has to be a programmer
  • Programming language doesn't always adequately represent testing concepts

"Hybrid" application testing

One type of scenario these tools do not cover well is hybrid situations where the test involves interactions between Java and native GUI components (such as web browsers containing an applet). In the Applet case, many or most of these tools can be used to automate tests of the Java GUI. However, they cannot automate interaction with the native GUI components. Here's the reasoning:

• The core of automating a GUI interaction is the question where to point the mouse. Where is the button/textfield/etc that you want to interact with? The first step in the interaction usually is to click on it.
• In Java it is pretty easy to traverse through the entire Component hierarchy. From any Java Component object you can find its screen coordinates (as required by Robot) simply through the getLocationOnScreen method.
• Java only lets the application see the Components in the current JVM. And if you hunt under the covers, you'll find some hurdles to seeing Components outside your current AppContext^? (important primarily for Applet's).
• The web browser GUI components or any other GUI component outside the current JVM instance are practically invisible to the test automation tool, preventing the test script from interacting with those components.

Obviously a commercial GUI test tool such as Silk,

WinRunner

RobotJ

Image-based component discovery

Most GUI test tools discover GUI components through some kind of hook or Java reflection. However this requires modifying the Java instance, and embedding the GUI test tool into the application may affect the overall execution. Further the technique limits component discovery to the components in the current VM, and if your test needs to interact with multiple applications then you'll be stuck.

There are at least two tools which follow a different approach. They perform component discovery through analyzing screen images. This lets them sit outside the application being tested, and to discover any component anywhere on the screen.

These are:

• XEUS: http://www.xeus-tool.com/ (commercial), written in Java, records the test script in XML, and bases all its discovery on image matching
• Eggplant: VNC based GUI testing http://www.redstonesoftware.com/
• VNCPlay: Interactive Session Replay and Performance Measurement http://suif.stanford.edu/vncplay/
• TestQuest: http://testquest.com/ (commercial), runs in a separate Windows-based PC and interfaces to the computer being tested through the video and keyboard/mouse ports.

Non-Java-specific GUI test automation tools

There are several GUI test automation tools which can drive non-Java applications that might also be useful to drive Java applications. Some are commercial, some open source. TBD: Find these and make links and summaries

• Silk (Segue)
• WinRunner^?

Articles & such

ClientJava.COM: Jemmy Swing GUI Testing - A presentation given in March 2005 at the St. Louis JUG meeting.

UI Automated Testing of Java Applications - Presentation given in June 2007 at Jazoon07 by Jiri Skrivanek

-- Main.robogeek - 12 Mar 2005

Discussion about TestingGUIApplications