Difference between revisions of "Testtool-feedback"

From Apache OpenOffice Wiki
Jump to: navigation, search
(How to use VCLTesttoool: added links to VCLTesttool documentation/manual/cookbook/quick-start guide)
 
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
[[Category:Quality Assurance]]
 +
 
Here the feedback on [[VCLTesttool]], on the test environment for VCLTesttool, on the test processes with VCLTesttool and tooling which will be used for analyze test results (e.g. [[QUASTe]]) should be used.  
 
Here the feedback on [[VCLTesttool]], on the test environment for VCLTesttool, on the test processes with VCLTesttool and tooling which will be used for analyze test results (e.g. [[QUASTe]]) should be used.  
  
 
== Feedback on VCLTesttool or simply TestTool ==
 
== Feedback on VCLTesttool or simply TestTool ==
 
This section should be used to collect feedback which is related to the VCLTesttool itself. Questions about test scripts should asked in the next section.
 
This section should be used to collect feedback which is related to the VCLTesttool itself. Questions about test scripts should asked in the next section.
 
+
=== How to use VCLTesttoool ===
* Description : How to use the VCLTesttool?
+
* Description : How to use VCLTesttool?
* Information : The VCLTesttool isn't a one-click solution. It is more complex and some background information are needed to run tests or to work with VCLTesttool. So please read the manual in the Automation project in OOo QA project : http://qa.openoffice.org/ooQAReloaded/AutomationTeamsite/ooQA-TeamAutomation.html
+
* Information : The VCLTesttool isn't a one-click solution. It is more complex and some background information are needed to run tests or to work with VCLTesttool. So please take a look on [http://qa.openoffice.org/ooQAReloaded/AutomationTeamsite/ooQA-TeamAutomation.html the Automation project on OOo]. There you can find links to the [http://qa.openoffice.org/qatesttool/OOo_testtool.pdf general documentation or manual of VCLTesttool] and a [http://qa.openoffice.org/qatesttool/OOo_tt_CookBook.pdf cookbook for writing test scripts] and a link to the [http://wiki.services.openoffice.org/wiki/VCLTesttool quick-start guide], which is up to date for OOo 3.x.
 
* Solution    : For a quick start guide for VCLTesttool read this : http://wiki.services.openoffice.org/wiki/VCLTesttool
 
* Solution    : For a quick start guide for VCLTesttool read this : http://wiki.services.openoffice.org/wiki/VCLTesttool
 
  
 
== Feedback on Test Environment ==
 
== Feedback on Test Environment ==
Line 30: Line 31:
 
This section should be used to collect feedback on QUASTe. [http://quaste.services.openoffice.org/ QUASTe] is the tooling/front end which can/should be used for analyzing the error-reports of the automated test runs.
 
This section should be used to collect feedback on QUASTe. [http://quaste.services.openoffice.org/ QUASTe] is the tooling/front end which can/should be used for analyzing the error-reports of the automated test runs.
  
=== entries for platform are missing ===
+
=== Entries for platform are missing ===
 
* Description : In QUASTe platforms like Mac PPC, Linux 64bit and other platforms which aren't fully supported by Sun are missing.
 
* Description : In QUASTe platforms like Mac PPC, Linux 64bit and other platforms which aren't fully supported by Sun are missing.
 
* Information : The system information taken automatically by VCLTesttool cannot recognize all different platforms. Therefore only the groups of platforms (Mac, Linux, Solaris Sparc, Solaris Intel) are used. Because of different version numbers it is possible to differ on Windows like Win98, WinXP and Vista. But it isn't possible for the VCLTesttool to identify an 64 Linux system or a Mac PPC automatically. So it isn't a problem of QUASTe it is a problem of VCLTesttool.
 
* Information : The system information taken automatically by VCLTesttool cannot recognize all different platforms. Therefore only the groups of platforms (Mac, Linux, Solaris Sparc, Solaris Intel) are used. Because of different version numbers it is possible to differ on Windows like Win98, WinXP and Vista. But it isn't possible for the VCLTesttool to identify an 64 Linux system or a Mac PPC automatically. So it isn't a problem of QUASTe it is a problem of VCLTesttool.
 
* Solution    : Currently there isn't a solution for this problem.
 
* Solution    : Currently there isn't a solution for this problem.

Latest revision as of 09:27, 19 December 2008


Here the feedback on VCLTesttool, on the test environment for VCLTesttool, on the test processes with VCLTesttool and tooling which will be used for analyze test results (e.g. QUASTe) should be used.

Feedback on VCLTesttool or simply TestTool

This section should be used to collect feedback which is related to the VCLTesttool itself. Questions about test scripts should asked in the next section.

How to use VCLTesttoool

Feedback on Test Environment

This section should be used to collect feedback for the test scripts and the test environment (running test scripts in different languages or on different platforms/operating systems etc.).

Test results differ to the references made by Sun

  • Description : My tests show more errors and warnings in a test result as the results which were made by Sun engineers.
  • Information : Sun used for automated testing on Master builds one dedicated system per platform. Most of the tests are optimized to run in the Sun environment and on such dedicated systems. This isn't possible in the community until now. Therefore system dependencies can be one reason for different results in specific tests. Other reasons can be that languages of OOo are tested which aren't fully supported in the test scripts. If a language isn't supported at all, the test should throw warning/errors with such a content or does not start at all. Another problem can be that builds from buildbots were used for testing. It isn't clear until now, which different build environments leads to which errors. Also other and until now unknown dependencies can be a reason for different results.
  • Solution  : Only in QUASTe you can get the information, if your test results differ from the references by Sun. Please write an issue in issue tracker for each of the differences. Address them to the script owner which is written in the header of the script or to Helge Delfs (HDE). If in a by Sun unsupported platform differences exists, please contact the script writer. It isn't clear, if the issue can be evaluated by the script writer or has to be debugged by the supporter of the platform. In every case the highest priority is to bring the scripts in the situation, that the results in external and Sun test environments are as equal as possible. Do not forget : In most cases the differences are bugs in OOo. Perhaps it is only triggered by VCLTesttool and not reproducible by manual testing. But then it is also a bug in OOo. But in most cases the fix of such a bug has less priority and therefore the test scripts will be adapted instead of fixing the bug.

One-Click-Solution for running a test script

  • Description : The VCLTesttool and the Test Scripts are different parts. It is needed to have one place where I can download one package to run a test script by one-click or to have one compact application to run tests.
  • Information : The VCLTesttool itself is part of OOo install sets and will be installed by default in the program directory. Therefore the tool is always up-to-date and match to the office version/build. The test scripts aren't part of the OOo install sets and are in change for each build of OOo. When new features are added or functionality in OOo changed the test scripts has to be adapted. If this isn't done, the tests will throw errors and warning where isn't a bug. So it is important to take always the current test scripts for testing the corresponding OOO build. Only then it is possible to get a correct quality status of OOo with VCLTesttool.
  • Solution  : There isn't a one-click-solution available until now and nobody worked on it. Because there isn't a clear picture of a such a solution. Currently the test scripts are be checked out automatically for each milestone. The packages can be downloaded from : http://ooopackages.good-day.net/pub/OpenOffice.org/qa/testautomation/


Feedback on Test Processes with VCLTesttool

This section should be used to collect feedback for processes where automated testing is integrated.


Feedback on QUASTe

This section should be used to collect feedback on QUASTe. QUASTe is the tooling/front end which can/should be used for analyzing the error-reports of the automated test runs.

Entries for platform are missing

  • Description : In QUASTe platforms like Mac PPC, Linux 64bit and other platforms which aren't fully supported by Sun are missing.
  • Information : The system information taken automatically by VCLTesttool cannot recognize all different platforms. Therefore only the groups of platforms (Mac, Linux, Solaris Sparc, Solaris Intel) are used. Because of different version numbers it is possible to differ on Windows like Win98, WinXP and Vista. But it isn't possible for the VCLTesttool to identify an 64 Linux system or a Mac PPC automatically. So it isn't a problem of QUASTe it is a problem of VCLTesttool.
  • Solution  : Currently there isn't a solution for this problem.
Personal tools