Revision as of 07:37, 6 April 2010

Test Cleanup Project

See the mission statement mail.

User Interface

CppUnit

For tests in C++, plain CppUnit (at least version 1.12.1) is available. (OOo bundles CppUnit 1.12.1 as an external module; alternatively, you can configure --with-system-cppunit.)

Implement the test code as a dynamic library, following CppUnit conventions (i.e., using CPPUNIT_TEST_SUITE_REGISTRATION and CPPUNIT_PLUGIN_IMPLEMENT macros). The following settings are needed in the makefile.mk:

• ENABLE_EXCEPTIONS = TRUE
• CFLAGSCXX += $(CPPUNIT_CFLAGS)
• SHLnRPATH = NONE
• SHLnSTDLIBS += $(CPPUNITLIB)
• SHLnVERSIONMAP = version.map, where that file must contain the following (see details below for improving this):

 UDK_3_0_0 {
   global:
     cppunitTestPlugIn;
   local:
     *;
 };

To run tests, sal/cppunittester delivers a cppunittester executable that has some advantages over the plain CppUnit DllPlugInTester:

• It uses SAL_IMPLEMENT_MAIN to properly set up the per-process settings required by URE. Test code as well as tested code based on URE will expect this and would fail in a process that is not set up in this way.
• Getting CppUnit to build as an external module within the Windows OOo build is a non-trivial task. The less parts of it that need to be built, the better.

Something like the following in makefile.mk will execute the tests:

 ALLTAR : test
 test .PHONY : $(SHL1TARGETN)
   $(CPPUNITTESTER) $(SHL1TARGETN)

$(CPPUNITTESTER) internally uses $(AUGMENT_LIBRARY_PATH_LOCAL) to favor library instances in the local output tree over ones in solver. Alternatively to the above, .INCLUDE: _cppunit.mk calls $(CPPUNITTESTER) on each of $(SHL1TARGETN), …, $(SHL10TARGETN) that is defined.

JUnit

For tests in Java, plain JUnit (at least version 4) is available. (OOo requires JUnit as a prerequisite, see junit.org for downloads; you can configure --with-junit=… if it is not automatically found; alternatively, you can configure --without-junit, disabling JUnit-based tests.)

subsequenttests

Some tests have build and/or run time dependencies on modules further up in the hierarchy, or modules that the rest of the module in question does not have dependencies on (and which are thus not listed in the module's build.lst). The extreme case are tests that need a complete OOo installation as their test environment.

For such tests, the whole makefile.mk should be wrapped in:

 .IF "$(OOO_SUBSEQUENT_TESTS)" == ""
 nothing .PHONY :
 .ELSE
 ...
 .END

(and the directory included in the module's build.lst).

The idea is to do a complete OOo build first (cd smoketestoo_native && build --all) and then build and execute those tests either selectively or wholesale.

Selectively building a specific such test directory can be done with cd … && OOO_SUBSEQUENT_TESTS=TRUE dmake.

To build and execute all these test directories, the tool subsequenttests (in solenv/bin) exists. It scans the build.lst files of all modules for makefile.mks that mention OOO_SUBSEQUENT_TESTS and simply executes dmake in all those directories, with OOO_SUBSEQUENT_TESTS set. It supports a -Pn switch to build n directories in parallel.

Helpers

A new module test shall collect reusable code snippets for C++ and Java tests.

For now, it concentrates on letting a test connect to an OOo instance via remote UNO: A C++ test class can hold a test::OfficeConnection as class member and, in its CppUnit setUp and tearDown functions, call OfficeConnection::setUp and OfficeConnection::tearDown, respectively. Tests obtain access to the OOo instance via OfficeConnection::getFactory. Similarly, a Java test class can use an org.openoffice.test.OfficeConnection instance, offering the same functionality as its C++ counterpart.

For this to work, three additional steps are necessary:

• First, in instsetoo_native, building an installation set for the openoffice product in the default language (en-US) in any package format now takes care to always also build an archive package format (tar.gz resp. zip) of that product.
• Second, building smoketestoo_native permanently installs that archive into the solver. (Except on Windows, where problems with resulting long pathnames prevent this; there, as a workaround, each test individually unzips the archive to some local temporary directory.)
• Third, a test that requires an OOo instance includes installationtest.mk in its makefile.mk. That file makes sure to call the CppUnit and JUnit test runners with the information necessary for the helper classes to start the correct OOo instance with a fresh UserInstallation and set up an URP pipe connection. The information is tunneled into the test runner executables as UNO bootstrap parameters.

unoapi Tests

Details

Working on CWS sb118  , CWS sb120  .

CppUnit 1.12.1

• Replace existing cppunit and testshl2 combo with latest CppUnit 1.12.1.
• OOo configure --with-system-cppunit (for now only if also --with-system-stl!); minimal required version for now arbitrarily 1.12.1.
• TODO: Upstream PATCH_FILES (windows.patch?!?).
• TODO: RPATH in generated libraries and executables.
• TODO: Improve CPPUNIT_PLUGIN_EXPORT and get rid of version maps (GCC exception problems?).
• TODO: cppunit/makefile.mk MY_LIBS = -lm Solaris hack.
• TODO: In general OOo build environment, no longer set CFLAGS etc. as environment variables; have package-specific OOO_package_CFLAGS etc. and combine those into CFLAGS etc. where necessary (see OOO_STLPORT_CXXFLAGS etc. in cppunit/makefile.mk).
• TODO: Why use cppunittester instead of DllPlugInTester? (Remove building/delivering DllPlugInTester.)
• TODO: Make CppUnit::assertion_traits<rtl::OUString> from smoketestoo_native/smoketest.cxx generally available.

testshl2

• solenv/inc_cppunit.mk/cppunit.mk: “adapted cppunit.mk to cppunittester and no longer require ‘dmake test’ to run tests”.
• TODO: DEV300_m68 introduced TESTSHL2LIB (which I removed again) and uses it in many qa makefiles (that I need to touch someday, anyway).

Unit Tests

• OOO_SUBSEQUENT_TESTS (build.lst lists local dependencies of those directories, so that modifying a module and then calling “OOO_SUBSEQUENT_TESTS=x build” rebuilds the module and executes the tests).
• TODO: If Build Environment 2.0 would build directly to solver (instead of local output tree/deliver), that would simplify tests that currently need to take care to test local libraries instead of solver ones (see, for example, OOO_TEST_PREFIX in the services.rdb of stoc/test/uriproc/makefile.mk).
• TODO: Constructs like “-env:UNO_…=$(my_file)…” would fail for problematic characters (spaces!) in paths.

Smoke Test

• Redesign smoketestoo_native to use CppUnit and pre-installed OOo.
• TODO: For smoketestoo_native/makefile.mk “-env:arg-env=$(OOO_LIBRARY_PATH_VAR)"$${{$(OOO_LIBRARY_PATH_VAR)+=$$$(OOO_LIBRARY_PATH_VAR)}}"” mechanism: osl_executeProcess ustrEnvironments members without “=”: all platforms; documentation; documentation of osl_executeProcess_WithRedicrectedIO.
• TODO: Dynamically create smoketestdoc.sxw?
• TODO: “changed tools/debug to never fall back from writing to shell to writing to file, and read DBGSV_INIT from rtl::Bootstrap; the latter causes problems if tools/debug is called from static code executing before SAL_MAIN has called osl_setCommandArgs → removed statics from code and added assertions to osl/process; various improvements to smoketests.cxx (incl. passing an appropriate dbgsv.ini for non-pro builds)”: Instead of -env:DBGSV_INIT= in smoketest.cxx (and supporting DBGSV_INIT as bootstrap variable), globally in the build environment let DBGSV_INIT environment variable point to a dbgsv.ini with error=coredump, to catch all assertions thrown during build time (incl. ones from tests like smoke test; see Issue 109142 ).
• TODO: -headless? (appears to still display windows on Windows and Mac OS X, however, see Issue 109052 )

Pre-Installed OOo

• For Sun Hamburg Release Engineering, take MWS archive installation sets from “S:” instead of solver: “take installation set from SHIPDRIVE if appropriate”.
• TODO: PKGFORMAT=installed instead of archive?

unoapi Tests

• Known issues: Issue 109517 , Issue 109518 , Issue 109643 , Issue 109917 .
• Missing tests: Issue 109564 .
• Sporadic failures: Issue 109728 , Issue 109770 , Issue 109916 , Issue 109939 , Issue 110253 .