QA/test automation guide

Introduction

VCLAuto is a Java library for OpenOffice UI/functional testing like VCLTesttool. VCLAuto can be used with JUnit. The project is under developing and will be released in Apache OpenOffice 4.0.

There are a lot of testing codes in the OO project, e.g. unit test, qadevooo and smoketestoo_native. Why do we need the library?

Generally, unit test code (pure junit/cppunit test) is executed in the build process before OO is created. It's low-level and used to verify if the source code is right before they are linked to a real product. It's a white-box testing. If you want to verify a function/method/interface is correct, put your test code in unit test. qadevooo & smoketestoo_native uses UNO API/Macro to perform testing. To run them, a real OO product must be available. It's middle-level and used to verify if UNO api and business model work correctly. It's a gray-box testing. User interaction is not involved in this testing. It can't fully simulate a user behavior or check if GUI is correct. For example, check if user can draw a shape by dragging, check if OO automatically capitalize the first letter after typing a word, check if a button's checked when user change the selection, etc. Vclauto is high-level black-box testing. It performs testing more like a real user. It generates keyboard/mouse events, does GUI actions and gets information from the GUI to validate the function.

Background

Actually VCLAuto is a java version of VCLTesttool, which connects to the automation server (automation module) in OpenOffice with socket. With the "-enableautomation" argument, OpenOffice will start the server with listening on the port 12479 be default. VCLTesttool is heavily used by the QA team in SUN/Oracle, but many people doesn't like it because of its drawbacks.

• The poor IDE. (Java has many enough powerful IDEs to make writing code easily, e.g. Eclipse / Netbeans)
• Hard to debug.
• Hard to read the code and analyze the testing result. (A lot of code is written in Non-English. Maybe German)
• Too many errors
• Basic language is not popular.

I found Test_Cleanup project starts to unify all testing code to follow the standard xUnit style, so I think it's time to clean up GUI test as well.

Getting Started with Eclipse

Setup Project

Get and install Eclipse Java IDE firstly. http://www.eclipse.org/downloads/

1. Open Eclipse, and then click menu "File->Import..."
2. Select "General->Existing Projects into Workspace", click next, and then select the following projects.
   

{AOO_SOURCE_ROOT}/test/testcommon

The project contains the common test utilities and low-level implementation to do GUI testing

{AOO_SOURCE_ROOT}/test/testgui

The project contains the GUI testing scripts. Test case should be written in this project.

1. Click "Finish" to import the projects.
2. Download JUnit 4.10+ jar from http://repo1.maven.org/maven2/junit/junit/4.10/junit-4.10.jar and put it into "testgui/output/lib/junit.jar"
3. Refresh the test project

Run Testing

Run Testing in Eclipse

1.Set AOO's installation location firstly. Open Eclipse->Preferences->Installed JREs->Select the checked JRE->Edit, set the system property "openoffice.home" int "Default VM Arguments"

 File:Openofficehome.png

2.Right click a test class. e.g. testcase.gui.SayHelloToOO, and select "Run As->JUnit Test" to start testing

Run Testing in Terminal

Firstly make sure Apache Ant 1.8.2+ installed. If not, download it from http://ant.apache.org/bindownload.cgi In the terminal, run command "ant -version" to verify the version.

How do I run testing on my OpenOffice?

cd testgui
ant -Dtest.classes="[Your_Test_Classes]" -Dopenoffice.home="[Your_OpenOffice_Location]"

How do I run build verification test after building AOO from source code?
Build verification test (BVT) is also known as Build Acceptance Test, is a set of tests run on each new build of a product to verify that the build is testable before the build is released into the hands of the test team. Currently, BVT includes the existing "smoke test" (testing with Macro) and some new GUI test cases. It takes about 1 hours.

source LinuxX86Env.Set.sh
cd testgui && ant

Where to get the testing result?

By default, the testing result is stored in "testgui/../testspace/output".

• "report" stores the result in HTML, like this.

• "result" stores XML result.
• "screenshot" stores the snapshot when test is failed.
• "logs" stores the detail log.

Actually, if you don't want the beautiful report, Ant is not required. Use java command to start it after compiling.
java -cp junit.jar:testcommon.jar:testscript/output/class org.junit.runner.JUnitCore [test class name]

Write Testing Classes

Write GUI testing case with JUnit4

package testcase;
 
import static org.junit.Assert.*;
import static testlib.AppUtil.*;
import static testlib.UIMap.*;
 
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
 
import testlib.CalcUtil;
import testlib.Log;
 
/**
 * Before running the testing class, you need specify the AOO location firstly with system property openoffice.home.
 * 
 * @author test
 *
 */
public class SayHelloToOO {
 
	/**
	 * TestCapture helps us to do
	 * 1. Take a screenshot when failure occurs.
	 * 2. Collect extra data when OpenOffice crashes.
	 */
	@Rule
	public Log LOG = new Log();
 
 
	@Before
	public void setUp() throws Exception {
		app.start(); // make sure that OpenOffice is started before testing
	}
 
	@After
	public void tearDown() throws Exception {
		app.close(); // close OpenOffice after testing is finished avoid contaminating other test cases
	}
 
	/**
	 * Implement the case
	 */
	@Test
	public void helloOO() {
		startcenter.menuItem("File->New->Spreadsheet").select();
		calc.waitForExistence(10, 3);
		typeKeys("Hello");
		assertEquals("Assert", "Hello", CalcUtil.getCellInput("A1"));
	}
 
}

org.openoffice.test.vcl.Tester implements some methods to post keyboard/mouse event to OS.

Tester.typeKeys("AB<enter>");
//type shortcuts
Tester.typeKeys("<ctrl a><ctrl c><ctrl v>");
//Perform mouse click on screen
Tester.click(200, 200);

The classes under package org.openoffice.test.vcl.widgets can be used to interact with VCL controls. Generally, we should specify control's Help ID to construct these classes. Then invoke other methods to perform user actions.

VclWindow startcenter = new VclWindow("FWK_HID_BACKINGWINDOW");
startcenter.click();
VclButton someButton = new VclButton("SC_HID_INSWIN_CALC");
someButton.click();
boolean checked = someButton.getText();
VclListBox someListBox = new VclListBox("some.listbox.id");
someListBox.select("Item1");
String selected = someListBox.getSelText();

Suggest to define UI controls centrally in testlib.UIMap. The advantage is that we can easily update our scripts when UI changes.

How to Get Help ID

Use VCL Test Assistant Eclipse Plugin
Install the plugin into your eclipse.
File:Testassistant.zip
Open Eclipse Preferences, find the page "VCL Test Assistant" and then specify OpenOffice installation directory.

Open "VCL Test Assistant->VCL Explorer" view.

Click Launch to start OpenOffice and then click Inspect.
Now, OpenOffice will pops up a floating windows titled "DisplayHID". Drag the gray circle to some controls and then release the button.

Go back to VCL Explorer, you will get the Help IDs.

If you want to use any control, left double click on the "Name" item of the control which you want to use, then add a name to it. Then you will find a UI control defined automatically in the last line of UIMap.java.(%EclipseWorkspaceDir%\test\testscript\src\testlib\UIMap.java). Then you can use the control by name you defined instead of using control ID.

Add a Test Case

• QA/vclauto/A step by step example
  

GUI Test Cases Writing Rules

When you want to commit test codes, you'd better verify all the codes with following rules.
1. Every test case should have verification point.
e.g. Test scenario: Inserting a table in text document.
After you inserting a table in the document, you have to verify whether the table is inserted successfully.
2. Make sure test codes are concise, readable and easy to maintain.
3. In VCLAuto, “sleep()” is usually not needed. Here is some special situation “sleep()” is needed.

• Open a file. If you want to do some operations based on a sample file or a new created file. “sleep()” is needed to wait for the file opened successfully.
  

4. Make sure test codes have no warnings.
5. When you add a new class, remember to add AOO license in front of the class.
6. Put same operations into “@Before” method to avoid codes redundant.
7. Use Ctrl+A, then Ctrl+Shift+F to format source code when you complete all the codes.
8. Test all the test codes on windows/mac/linux platforms before committing.