Test Refactor

The page is created for tracking test refactor project.

Background

Test automation is important for Apache OpenOffice quality. Currently, there are many modules for test automation. Note, the mentioned testing here requires a running OpenOffice instances. That means OpenOffice must be installed before testing. It's not the unit test at the code level.

Module	Description	Comment
testgraphical	Test tool to test documents by it's graphical representation.
testautomation	All test scripts for the old VCL Testtool. Nobody maintains it now.
smoketestdoc	It's used to generate the test documents required by smoke test.
test	Includes reusable code for UNO API test.
smoketestoo_native	A small test suite to verify if the basic functions of OpenOffice.org work.
qadevOOo	UNO API test, complex test.
{Module}/qa	UNO API test, complex test.

The testing code can be categorized to 3 levels.

• Generally, unit test code (pure junit/cppunit test) is executed in the build process before OpenOffice is created. It's low-level and used to verify if the source code is right before they are linked to a real product. It's a white-box testing. If you want to verify a function/method/interface is correct, put your test code in unit test.
  
• qadevooo & smoketestoo_native uses UNO API/Macro to perform testing. To run them, a real OpenOffice product must be available. It's middle-level and used to verify if UNO api and business model work correctly. It's a gray-box testing. User interaction is not involved in this testing. It can't fully simulate a user behavior or check if GUI is correct. For example, check if user can draw a shape by dragging, check if OpenOffice automatically capitalize the first letter after typing a word, check if a button's checked when user change the selection, etc.
  
• VclTesttool is high-level black-box testing. It performs testing more like a real user. It generates keyboard/mouse events, does GUI actions and gets information from the GUI to validate the function.
  

Problems

The test code is organized in a mess and out of maintain. Even the most are not stable enough to give useful results. It's hard to get started. Three programming languages (Java, Basic and C++) are used. It's not necessary. VCLTesttool is heavily used by the QA team in SUN/Oracle, but many people doesn't like it because of its drawbacks.

• The poor IDE. (Java has many enough powerful IDEs to make writing code easily, e.g. Eclipse / Netbeans)
• Hard to debug.
• Hard to read the code and analyze the testing result. (A lot of code is written in Non-English. Maybe German)
• Too many errors
• Basic language is not popular.

Some contributors have cleaned up the test code, see http://wiki.services.openoffice.org/wiki/Test_Cleanup, but it seems the project is not finished yet.

Target

The target is to reduce module count, organize the testing code clearer, make test code enough stable to integrate with buildbot, and provide more readable test report. Propose to write all test based on JUnit 4, because Java is easier than C/C++, more popular than Basic. Supporting multiple languages need more effort to maintain and leads duplicate more easily. Test code should be broken down into three categories.

1. purely GUI test. It requires graphical user interfaces enabled. This kind of test requires simulating user interaction (posting keyboard/mouse events and accessing GUI).So it can not be performed in headless mode and hard to run in parallel.
2. purely API test. Graphical user interfaces is not required. We can run this kind of test in parallel to save time.
3. Mixed test. This kind of test involves both GUI test and API test. It also can not be parallelized. Generally, I think it's not necessary to mix them.

Suggest to move all test stuff into test module and create several sub-projects in it:

1. test/testcommon. The reusable code for both GUI test and API test. It doesn't depend on any project.
2. test/testuno. API test should implemented in the sub-project.
3. test/testgui. GUI test should implemented in the sub-project.

Developers can still write test code under {MODULE}/qa.

Under test module, we provide Ant script (or make file) to install build, start testing and then generate test report:

 cd test
 ant

We can select a part of test classes to run (It's useful when we only want to test the impact area):

 ant -Dtest.classes=testcase.uno.*

TODO

1. Create testcommon. Done!
2. Create testgui. Done!
3. Create testuno.. Done!
4. Move the code under test module to testcommon.
5. Convert testgraphical to JUnit4 test case.
6. Convert the test scripts in testautomation to testgui.
7. Convert smoketestdoc & smoketestoo_native to JUnit 4 test cases. Done!
8. Convert the test scripts in qadevooo into testuno.
9. Convert the test scripts in ${MODULE}/qa to JUnit 4 test cases.
10. Remove the old code under test, testgraphical, testautomation, smoketestdoc, smoketestoo_native and qadevooo.