Testtool-feedback

Here the feedback on VCL-TestTool, on the test environment for VCL-TestTool, on the test processes with VCL-TestTool and tooling which will be used for analyze test results (e.g. QUASTe) should be used.

Feedback on VCL-TestTool or simply TestTool

This section should be used to collect feedback which is related to the TestTool itself. Questions about test scripts should asked in the next section.

• Description : Who to use the TestTool?
• Information : The TestTool isn't a one-click solution. It is more complex and some background information are needed to run tests or to work with TestTool. So please read the manual in the Automation project in OOo QA project : http://qa.openoffice.org/ooQAReloaded/AutomationTeamsite/ooQA-TeamAutomation.html
• Solution  : For a quick start guide for TestTool read this : http://wiki.services.openoffice.org/wiki/VCLTesttool

Feedback on Test Environment

This section should be used to collect feedback for the test scripts and the test environment (running test scripts in different languages or on different platforms/operating systems etc.).

Test results differ to the references made by Sun

• Description : My tests show more errors and warnings in a test result as the results which were made by Sun engineers.
• Information : Sun used for automated testing on Master builds one dedicated system per platform. Most of the tests are optimized to run in the Sun environment and on such dedicated systems. This isn't possible in the community until now. Therefore system dependencies can be one reason for different results in specific tests. Other reasons can be that languages of OOo are tested which aren't fully supported in the test scripts. If a language isn't supported at all, the test should throw warning/errors with such a content or does not start at all. Another problem can be that builds from buildbots were used for testing. It isn't clear until now, which different build environments leads to which errors. Also other and until now unknown dependencies can be a reason for different results.
• Solution  : Only in QUASTe you can get the information, if your test results differ from the references by Sun. Please write an issue in issue tracker for each of the differences. Address them to the script owner which is written in the header of the script or to Helge Delfs (HDE). If in a by Sun unsupported platform differences exists, please contact the script writer. It isn't clear, if the issue can be evaluated by the script writer or has to be debugged by the supporter of the platform. In every case the highest priority is to bring the scripts in the situation, that the results in external and Sun test environments are as equal as possible. Do not forget : In most cases the differences are bugs in OOo. Perhaps it is only triggered by TestTool and not reproducible by manual testing. But then it is also a bug in OOo. But in most cases the fix of such a bug has less priority and therefore the test scripts will be adapted instead of fixing the bug.

One-Click-Solution for running a test script

• Description : The TestTool and the Test Scripts are different parts. It is needed to have one place where I can download one package to run a test script by one-click or to have one compact application to run tests.
• Information : The TestTool itself is part of OOo install sets and will be installed by default in the program directory. Therefore the tool is always up-to-date and match to the office version/build. The test scripts aren't part of the OOo install sets and are in change for each build of OOo. When new features are added or functionality in OOo changed the test scripts has to be adapted. If this isn't done, the tests will throw errors and warning where isn't a bug. So it is important to take always the current test scripts for testing the corresponding OOO build. Only then it is possible to get a correct quality status of OOo with TestTool.
• Solution  : There isn't a one-click-solution available until now and nobody worked on it. Because there isn't a clear picture of a such a solution. Currently the test scripts are be checked out automatically for each milestone. The packages can be downloaded from : http://ooopackages.good-day.net/pub/OpenOffice.org/qa/testautomation/

Feedback on Test Processes with TestTool

This section should be used to collect feedback for processes where automated testing is integrated.

Feedback on QUASTe

This section should be used to collect feedback on QUASTe. QUASTe is the tooling/front end which can/should be used for analyzing the error-reports of the automated test runs.

entries for platform are missing

• Description : In QUASTe platforms like Mac PPC, Linux 64bit and other platforms which aren't fully supported by Sun are missing.
• Information : Currently only platforms are in QUASTe which are fully supported by Sun. It isn't a problem to add other platform in this tooling. But the consequence is, that references (results of a test run on a master build in one language - e.g. English) have to be created for each milestone. Only then it makes sense to make comparison with other platforms, languages and CWSs based on this platform.
• Solution  : If somebody takes the responsibility for new platform in QUASTe the new platform will/can be integrated. Write an issue to Helge Delfs (HDE) with the naming of the platform.