Difference between revisions of "QA/test automation guide"

From Apache OpenOffice Wiki
< QA
Jump to: navigation, search
(→‎Prerequisites)
(47 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
== Introduction ==
+
{{DISPLAYTITLE: Getting Started with Test Automation}}
 +
[[Category: Quality Assurance]]
 +
[[Category: TestAutomation]]
  
−
VCLAuto is a Java library for OpenOffice UI/functional testing like [[VCLTesttool]].
 
−
VCLAuto can be used with JUnit. The project is under developing and  will be released in Apache OpenOffice 4.0.
 
  
−
'''There are a lot of testing codes in the OO project, e.g. unit test, qadevooo and smoketestoo_native. Why do we need the library?'''
+
== Introduction ==
  
−
Generally, unit test code (pure junit/cppunit test) is executed in the build process before OO is created. It's low-level and used to verify if the source code is right before they are linked to a real product. It's a white-box testing. If you want to verify a function/method/interface is correct, put your test code in unit test.
+
Two kind of approaches is supported by Apache OpenOffice.
−
qadevooo & smoketestoo_native uses UNO API/Macro to perform testing. To run them, a real OO product must be available. It's middle-level and used to verify if UNO api and business model work correctly. It's a gray-box testing. User interaction is not involved in this testing. It can't fully simulate a user behavior or check if GUI is correct. For example, check if user can draw a shape by dragging, check if OO automatically capitalize the first letter after typing a word, check if a button's checked when user change the selection, etc.
+
*GUI Testing: Simulating a common user to perform testing. It depends on a Java library named VCLAuto, which does testing like the old [[VCLTesttool]]. It can easily integrates with JUnit. VCLAuto provides APIs to generate user interface events such as keystrokes and mouse clicks, do GUI actions and get information from the GUI to validate functions.
−
Vclauto is high-level black-box testing. It performs testing more like a real user.  It generates keyboard/mouse events, does GUI actions and gets information from the GUI to validate the function.
+
*UNO API Testing:  Use UNO API to verify functions.
  
−
'''Background'''
+
== Notice ==
−
 
+
There are some reasons for test failures, so please [[QA/test_env|prepare your testing machine]] before running test.
−
Actually VCLAuto is a java version of  [[VCLTesttool]], which connects to the automation server (automation module) in OpenOffice with socket. With the "-enableautomation" argument, OpenOffice will start the server with listening on the port 12479 be default. 
+
−
[[VCLTesttool]] is heavily used by the QA team in SUN/Oracle, but many people doesn't like it because of its drawbacks.
+
−
*The poor IDE.  (Java has many enough powerful IDEs to make writing code easily, e.g. Eclipse / Netbeans)
+
−
*Hard to debug.
+
−
*Hard to read the code and analyze the testing result. (A lot of code is written in Non-English. Maybe German)
+
−
*Too many errors
+
−
*Basic language is not popular.
+
−
 
+
−
I found [[Test_Cleanup]] project starts to unify all testing code to follow the standard xUnit style, so I think it's time to clean up GUI test as well.
+
  
 
== Getting the source code ==
 
== Getting the source code ==
 
Anyone can checkout source code from our Subversion repository. Run the following command.
 
Anyone can checkout source code from our Subversion repository. Run the following command.
−
svn co https://svn.apache.org/repos/asf/incubator/ooo/trunk/main/test/ test
+
<source lang="bash">
−
If you are not familiar with Subversion, see our [http://incubator.apache.org/openofficeorg/svn-basics.html Subversion Basics] for more information.
+
svn co https://svn.apache.org/repos/asf/openoffice/trunk/test/ test
 +
</source>
 +
If you are not familiar with Subversion, see our [http://openoffice.apache.org/svn-basics.html Subversion Basics] for more information.
 +
 
 
== Getting started with Eclipse ==
 
== Getting started with Eclipse ==
 
=== Prerequisites ===  
 
=== Prerequisites ===  
 
*Eclipse Java IDE: [http://www.eclipse.org/downloads/packages/eclipse-ide-java-developers/junor Eclipse Juno(4.2) IDE for Java Developers] or above.<br />
 
*Eclipse Java IDE: [http://www.eclipse.org/downloads/packages/eclipse-ide-java-developers/junor Eclipse Juno(4.2) IDE for Java Developers] or above.<br />
 +
*Apache OpenOffice
 +
 
=== Setup project ===
 
=== Setup project ===
−
Step 1. Open Eclipse, click menu "File->Import...", and then select "General->Existing Projects into Workspace".<br />
+
'''Step 1.''' Open Eclipse, click menu "File->Import...", and then select "General->Existing Projects into Workspace".<br />
−
[[File:vclauto_guide_1.png]]<br />
+
[[File:test_automation_guide_1.png]]<br />
−
Step 2. Click next, set "Select root directory" to the source code directory, and then check the following projects.<br />
+
'''Step 2.''' Click next, set "Select root directory" to the source code directory, and then check the following projects.<br />
−
  '''testcommon''': The project contains the common test utilities and low-level implementation to do GUI testing
+
  '''testcommon''': The project includes the common test utilities and low-level implementation to do GUI testing
−
  '''testgui''':  The project contains the GUI testing scripts. Test case should be written in this project.
+
  '''testgui''':  The project includes the GUI testing scripts.  
−
[[File:vclauto_guide_2.png]]<br />
+
'''testuno''':  The project includes the UNO API testing scripts.
−
Step 3. Click "Finish" to import the projects.
+
[[File:test_automation_guide_2.png]]<br />
 +
'''Step 3.''' Click "Finish" to import the projects.
 +
 
 +
'''Step 4.''' Click menu "Window->Preferences" ("Eclipse->Preferences" on Mac) and then select "Java->Build Path->Classpath Variables" page.<br />
 +
[[File:test_automation_guide_10.png]]<br />
 +
Click "New" to create one new variable "'''openoffice.home'''", and set its value to Apache OpenOffice installation directory.<br />
 +
[[File:test_automation_guide_11.png]]<br />
 +
{{Note| '''The default installation directory of Apache OpenOffice'''<br />
 +
Mac OS X: "''/Applications/OpenOffice.app/Contents"''<br />
 +
Linux: "''/opt/openoffice3''"<br />
 +
Windows: "''C:/Program Files/OpenOffice 3''"<br />
 +
Windows 64b: "''C:/Program Files (x86)/OpenOffice 3''"
 +
}}
 +
Click "OK" to close "New Variable Entry"dialog. Then click "OK" button of preferences dialog. A message box pops up. Click "Yes".<br />
 +
[[File:test_automation_guide_12.png]]<br />
 +
 
 
=== Run testing ===
 
=== Run testing ===
−
Step 1. Set AOO's installation location firstly. If AOO is installed in the default directory, you can skip this step. <br />
+
'''Step 1.''' Select one test class. e.g. testcase.gui.AOOTest, and then click "Run" on the toolbar to start testing. <br />
−
Otherwise, click menu "Window->Preferences" ("Eclipse->Preferences" on Mac), select "Java->Installed JREs" page and then select the checked JRE.<br />
+
[[File:test_automation_guide_13.png]]<br />
−
[[File:vclauto_guide_10.png]]<br />
+
−
Click "Edit" button and then add the following line in "Default VM Arguments"<br />
+
−
-Dopenoffice.home="Your OpenOffice installation directory which contains soffice.bin"
+
−
[[File:vclauto_guide_12.png]]<br />
+
−
Click "Finish".<br />
+
−
Step 2. Select one test class. e.g. testcase.gui.SayHelloToOO, and then click "Run" on the toolbar to start testing<br />
+
−
[[File:vclauto_guide_14.png]]<br />
+
 
When testing is finished, JUnit view will show the result.<br />
 
When testing is finished, JUnit view will show the result.<br />
−
[[File:vclauto_guide_16.png]]<br />
+
[[File:test_automation_guide_14.png]]<br />
 +
You also can select one package, click "Run", and then choose "JUnit test" to run all test classes under it.
  
 
== Getting started with command line ==
 
== Getting started with command line ==
Line 58: Line 63:
  
 
=== Run testing  ===
 
=== Run testing  ===
−
'''How do I run testing on OpenOffice?'''
+
'''As a developer, how to run testing against an OpenOffice built by me?''' <br />
−
Run the following commands.
+
It's easy for developers to run testing after building OpenOffice. One thing you need to do is to make sure you run the commands in the build environment. The script will automatically install your build and then start testing. e.g.
 
<source lang="bash">
 
<source lang="bash">
−
cd test
+
cd main
−
ant -Dtest.classes="Test class list" -Dopenoffice.home="Your OpenOffice installation directory which contains soffice.bin"
+
source MacOSXX86Env.Set.sh
 +
cd ../test && ant
 
</source>
 
</source>
−
e.g. Run testcase/gui/SayHelloToOO and testcase/gui/SmokeTest against OpenOffice installed in "/home/test/OpenOffice.org.app"
+
 
 +
'''As a normal user, how to run testing against an installed OpenOffice?'''<br />
 +
Firstly compile the project with the command under test module.
 
<source lang="bash">
 
<source lang="bash">
−
cd test
+
ant -Dopenoffice.home="OpenOffice installation directory" compile
−
ant -Dtest.classes="testcase/gui/SayHelloToOO.class,testcase/gui/SmokeTest.class" -Dopenoffice.home="/home/test/OpenOffice.org.app/Contents/MacOS"
+
 
</source>
 
</source>
−
If test.classes is not specified, BVT will be executed. If openoffice.home is not specified,  it will try to test OpenOffice installed in the default directory. e.g. Run BVT against OpenOffice  in the default directory.
+
Then start testing with the command "run".  
 +
e.g. Run all test classes under package bvt ("Build Verification Testing")
 
<source lang="bash">
 
<source lang="bash">
−
cd test
+
run -Dopenoffice.home="/Applications/OpenOffice.app/Contents" -tp bvt
−
ant
+
 
</source>
 
</source>
−
{{Template:Documentation/Note| '''The default value of openoffice.home'''<br />
+
e.g. Run the given test classes
−
Mac OS X: "''/Applications/OpenOffice.org.app/Contents/MacOS"''<br />
+
−
Linux: "''/opt/openoffice.org3/program''"<br />
+
−
Windows: "''C:/Program Files/OpenOffice.org 3/program''"<br />
+
−
Windows 64b: "''C:/Program Files (x86)/OpenOffice.org 3/program''"
+
−
}}
+
−
 
+
−
 
+
−
'''How do I run build verification test after building AOO from source code?''' <br />
+
−
Build verification test (BVT) is also known as Build Acceptance Test, is a set of tests run on each new build of a product to verify that the build is testable before the build is released into the hands of the test team<ref>http://en.wikipedia.org/wiki/Build_verification_test</ref>. Currently, BVT includes the existing "smoke test" (testing with Macro) and some new GUI test cases. It takes about 1 hours.
+
 
<source lang="bash">
 
<source lang="bash">
−
source LinuxX86Env.Set.sh
+
run -Dopenoffice.home="/Applications/OpenOffice.app/Contents" -tc bvt.gui.BasicFunctionTest
−
cd test && ant
+
 
</source>
 
</source>
 +
On Linux use e.g. -Dopenoffice.home="/opt/openoffice4/program" instead.
 +
  
 
''' Where to get the testing result? '''
 
''' Where to get the testing result? '''
  
−
By default, the testing result is stored in "testgui/../testspace/output".
+
By default, the testing output is stored in "test/testspace/output***". Open "test/testspace/output/result.html" in your browser to see the testing report.<br />
−
*"report" stores the result in HTML, like this.
+
[[File:test_automation_report.png]]<br />
−
[[File:vclautoreport.png]]
+
−
*"result" stores XML result.
+
−
*"screenshot" stores the snapshot when test is failed.
+
−
*"logs" stores the detail log.
+
−
<br />
+
−
Actually, if you don't want the beautiful report, Ant is not required. Use java command to start it after compiling.<br />
+
−
java -cp junit.jar:testcommon.jar:testscript/output/class org.junit.runner.JUnitCore [test class name]
+
−
 
+
−
== Write Testing Classes ==
+
−
 
+
−
=== Write GUI testing case with JUnit4 ===
+
−
 
+
−
<source lang="java">
+
−
package testcase;
+
−
 
+
−
import static org.junit.Assert.*;
+
−
import static testlib.AppUtil.*;
+
−
import static testlib.UIMap.*;
+
−
 
+
−
import org.junit.Before;
+
−
import org.junit.Rule;
+
−
import org.junit.Test;
+
−
 
+
−
import testlib.CalcUtil;
+
−
import testlib.Log;
+
−
 
+
−
/**
+
−
* Before running the testing class, you need specify the AOO location firstly with system property openoffice.home.
+
−
*
+
−
* @author test
+
−
*
+
−
*/
+
−
public class SayHelloToOO {
+
−
 
+
−
/**
+
−
* TestCapture helps us to do
+
−
* 1. Take a screenshot when failure occurs.
+
−
* 2. Collect extra data when OpenOffice crashes.
+
−
*/
+
−
@Rule
+
−
public Log LOG = new Log();
+
−
 
+
−
 
+
−
@Before
+
−
public void setUp() throws Exception {
+
−
app.start(); // make sure that OpenOffice is started before testing
+
−
}
+
−
 
+
−
@After
+
−
public void tearDown() throws Exception {
+
−
app.close(); // close OpenOffice after testing is finished avoid contaminating other test cases
+
−
}
+
−
 
+
−
/**
+
−
* Implement the case
+
−
*/
+
−
@Test
+
−
public void helloOO() {
+
−
startcenter.menuItem("File->New->Spreadsheet").select();
+
−
calc.waitForExistence(10, 3);
+
−
typeKeys("Hello");
+
−
assertEquals("Assert", "Hello", CalcUtil.getCellInput("A1"));
+
−
}
+
−
 
+
−
}
+
−
 
+
−
</source>
+
−
 
+
−
org.openoffice.test.vcl.Tester implements some methods to post keyboard/mouse event to OS.
+
−
<source lang="java">
+
−
Tester.typeKeys("AB<enter>");
+
−
//type shortcuts
+
−
Tester.typeKeys("<ctrl a><ctrl c><ctrl v>");
+
−
//Perform mouse click on screen
+
−
Tester.click(200, 200);
+
−
</source>
+
−
 
+
−
The classes under package org.openoffice.test.vcl.widgets can be used to interact with VCL controls. Generally, we should specify control's Help ID to construct these classes. Then invoke other methods to perform user actions.
+
−
<source lang="java">
+
−
VclWindow startcenter = new VclWindow("FWK_HID_BACKINGWINDOW");
+
−
startcenter.click();
+
−
VclButton someButton = new VclButton("SC_HID_INSWIN_CALC");
+
−
someButton.click();
+
−
boolean checked = someButton.getText();
+
−
VclListBox someListBox = new VclListBox("some.listbox.id");
+
−
someListBox.select("Item1");
+
−
String selected = someListBox.getSelText();
+
−
</source>
+
−
Suggest to define UI controls centrally in testlib.UIMap. The advantage is that we can easily update our scripts when UI changes.<br />
+
−
 
+
−
=== How to Get Help ID ===
+
−
''' Use VCL Test Assistant Eclipse Plugin '''<br />
+
−
Install the plugin into your eclipse.<br />
+
−
[[File:testassistant.zip]] <br />
+
−
Open Eclipse Preferences, find the page "VCL Test Assistant" and then specify OpenOffice installation directory.<br />
+
−
[[File:vcl_test_assistant_preference.png]]<br />
+
−
Open "VCL Test Assistant->VCL Explorer" view.<br />
+
−
[[File:vcl_test_assistant_view.png]]<br />
+
−
Click Launch to start OpenOffice and then click Inspect. <br />
+
−
Now, OpenOffice will pops up a floating windows titled "DisplayHID". Drag the gray circle to some controls and then release the button. <br />
+
−
[[File:inspect_help_id.png]]<br />
+
−
Go back to VCL Explorer, you will get the Help IDs.<br />
+
−
[[File:help_id_view.png]]<br />
+
−
If you want to use any control, left double click on the "Name" item of the control which you want to use, then add a name to it. Then you will find a UI control defined automatically in the last line of UIMap.java.(%EclipseWorkspaceDir%\test\testscript\src\testlib\UIMap.java). Then you can use the control by name you defined instead of using control ID.
+
−
[[File:Add_controls.png]]<br />
+
  
−
=== Add a Test Case ===
+
Special stuff in the output directory.
−
* [[QA/vclauto/A step by step example]]<br />
+
result.xml: Test result in HTML.
 +
result.html: Test result in XML.
 +
screenshot/: Screenshot pictures when test assert is failed.
  
−
=== GUI Test Cases Writing Rules ===
+
== Next ==
−
When you want to commit test codes, you'd better verify all the codes with following rules.<br />
+
Now, all things have been ready for developing new test code. If you are interested, please read [[QA/dev_gui_testing|GUI Testing Development]] and [[QA/dev_api_testing|UNO API Testing Development]] to get more information.
−
1. Every test case should have verification point.<br />
+
−
e.g. Test scenario: Inserting a table in text document.<br />
+
−
After you inserting a table in the document, you have to verify whether the table is inserted successfully.<br />
+
−
2. Make sure test codes are concise, readable and easy to maintain.<br />
+
−
3. In VCLAuto, “sleep()” is usually not needed. Here is some special situation “sleep()” is needed.<br />
+
−
* Open a file. If you want to do some operations based on a sample file or a new created file. “sleep()” is needed to wait for the file opened successfully.<br />
+
−
4. Make sure test codes have no warnings.<br />
+
−
5. When you add a new class, remember to add AOO license in front of the class.<br />
+
−
6. Put same operations into “@Before” method to avoid codes redundant.<br />
+
−
7. Use Ctrl+A, then Ctrl+Shift+F to format source code when you complete all the codes.<br />
+
−
8. Test all the test codes on windows/mac/linux platforms before committing.<br />
+

Revision as of 13:55, 7 July 2018


Introduction

Two kind of approaches is supported by Apache OpenOffice.

  • GUI Testing: Simulating a common user to perform testing. It depends on a Java library named VCLAuto, which does testing like the old VCLTesttool. It can easily integrates with JUnit. VCLAuto provides APIs to generate user interface events such as keystrokes and mouse clicks, do GUI actions and get information from the GUI to validate functions.
  • UNO API Testing: Use UNO API to verify functions.

Notice

There are some reasons for test failures, so please prepare your testing machine before running test.

Getting the source code

Anyone can checkout source code from our Subversion repository. Run the following command. 

svn co https://svn.apache.org/repos/asf/openoffice/trunk/test/ test

If you are not familiar with Subversion, see our Subversion Basics for more information.

Getting started with Eclipse

Prerequisites

Setup project

Step 1. Open Eclipse, click menu "File->Import...", and then select "General->Existing Projects into Workspace".
Test automation guide 1.png
Step 2. Click next, set "Select root directory" to the source code directory, and then check the following projects.

testcommon: The project includes the common test utilities and low-level implementation to do GUI testing
testgui:  The project includes the GUI testing scripts. 
testuno:  The project includes the UNO API testing scripts.

Test automation guide 2.png
Step 3. Click "Finish" to import the projects.

Step 4. Click menu "Window->Preferences" ("Eclipse->Preferences" on Mac) and then select "Java->Build Path->Classpath Variables" page.
Test automation guide 10.png
Click "New" to create one new variable "openoffice.home", and set its value to Apache OpenOffice installation directory.
Test automation guide 11.png

Documentation note.png The default installation directory of Apache OpenOffice
Mac OS X: "/Applications/OpenOffice.app/Contents"

Linux: "/opt/openoffice3"

Windows: "C:/Program Files/OpenOffice 3"

Windows 64b: "C:/Program Files (x86)/OpenOffice 3"

Click "OK" to close "New Variable Entry"dialog. Then click "OK" button of preferences dialog. A message box pops up. Click "Yes".
Test automation guide 12.png

Run testing

Step 1. Select one test class. e.g. testcase.gui.AOOTest, and then click "Run" on the toolbar to start testing.
Test automation guide 13.png
When testing is finished, JUnit view will show the result.
Test automation guide 14.png
You also can select one package, click "Run", and then choose "JUnit test" to run all test classes under it.

Getting started with command line

Prerequisites

Run testing

As a developer, how to run testing against an OpenOffice built by me?
It's easy for developers to run testing after building OpenOffice. One thing you need to do is to make sure you run the commands in the build environment. The script will automatically install your build and then start testing. e.g. 

cd main
source MacOSXX86Env.Set.sh
cd ../test && ant

As a normal user, how to run testing against an installed OpenOffice?
Firstly compile the project with the command under test module. 

ant -Dopenoffice.home="OpenOffice installation directory" compile

Then start testing with the command "run". e.g. Run all test classes under package bvt ("Build Verification Testing") 

run -Dopenoffice.home="/Applications/OpenOffice.app/Contents" -tp bvt

e.g. Run the given test classes 

run -Dopenoffice.home="/Applications/OpenOffice.app/Contents" -tc bvt.gui.BasicFunctionTest

On Linux use e.g. -Dopenoffice.home="/opt/openoffice4/program" instead.


Where to get the testing result?

By default, the testing output is stored in "test/testspace/output***". Open "test/testspace/output/result.html" in your browser to see the testing report.
File:Test automation report.png

Special stuff in the output directory. 

result.xml: Test result in HTML. 
result.html: Test result in XML.
screenshot/: Screenshot pictures when test assert is failed.

Next

Now, all things have been ready for developing new test code. If you are interested, please read GUI Testing Development and UNO API Testing Development to get more information.

Retrieved from "https://wiki.openoffice.org/w/index.php?title=QA/test_automation_guide&oldid=241217"
Personal tools