Revision as of 11:43, 26 April 2010

Test Cleanup Project

See the mission statement mail and slides from a FOSDEM'10 talk on the subject.

User Interface

CppUnit

For tests in C++, plain CppUnit (at least version 1.12.1) is available. (OOo bundles CppUnit 1.12.1 as an external module; alternatively, you can configure --with-system-cppunit.)

Implement the test code as a dynamic library, following CppUnit conventions (i.e., using CPPUNIT_TEST_SUITE_REGISTRATION and CPPUNIT_PLUGIN_IMPLEMENT macros). The following settings are needed in the makefile.mk:

• ENABLE_EXCEPTIONS = TRUE
• CFLAGSCXX += $(CPPUNIT_CFLAGS)
• SHLnRPATH = NONE
• SHLnSTDLIBS += $(CPPUNITLIB)
• SHLnVERSIONMAP = version.map, where that file must contain the following (see details below for improving this):

 UDK_3_0_0 {
   global:
     cppunitTestPlugIn;
   local:
     *;
 };

To run tests, sal/cppunittester delivers a cppunittester executable that has some advantages over the plain CppUnit DllPlugInTester:

• It uses SAL_IMPLEMENT_MAIN to properly set up the per-process settings required by URE. Test code as well as tested code based on URE will expect this and would fail in a process that is not set up in this way.
• Getting CppUnit to build as an external module within the Windows OOo build is a non-trivial task. The less parts of it that need to be built, the better.

Something like the following in makefile.mk will execute the tests:

 ALLTAR : test
 test .PHONY : $(SHL1TARGETN)
   $(CPPUNITTESTER) $(SHL1TARGETN)

$(CPPUNITTESTER) internally uses $(AUGMENT_LIBRARY_PATH_LOCAL) to favor library instances in the local output tree over ones in solver. Alternatively to the above, .INCLUDE: _cppunit.mk calls $(CPPUNITTESTER) on each of $(SHL1TARGETN), …, $(SHL10TARGETN) that is defined.

JUnit

For tests in Java, plain JUnit (at least version 4) is available. (OOo requires JUnit as a prerequisite, see junit.org for downloads; you can configure --with-junit=… if it is not automatically found; alternatively, you can configure --without-junit, disabling JUnit-based tests.)

subsequenttests

Some tests have build and/or run time dependencies on modules further up in the hierarchy, or modules that the rest of the module in question does not have dependencies on (and which are thus not listed in the module's build.lst). The extreme case are tests that need a complete OOo installation as their test environment.

For such tests, the whole makefile.mk should be wrapped in:

 .IF "$(OOO_SUBSEQUENT_TESTS)" == ""
 nothing .PHONY :
 .ELSE
 ...
 .END

(and the directory included in the module's build.lst).

The idea is to do a complete OOo build first (cd smoketestoo_native && build --all) and then build and execute those tests either selectively or wholesale.

Selectively building a specific such test directory can be done with cd … && OOO_SUBSEQUENT_TESTS=TRUE dmake.

To build and execute all these test directories, the tool subsequenttests (in solenv/bin) exists. It scans the build.lst files of all modules for makefile.mks that mention OOO_SUBSEQUENT_TESTS and simply executes dmake in all those directories, with OOO_SUBSEQUENT_TESTS set. It supports a -Pn switch to build n directories in parallel.

Helpers

A new module test shall collect reusable code snippets for C++ and Java tests.

For now, it concentrates on letting a test connect to an OOo instance via remote UNO: A C++ test class can hold a test::OfficeConnection as class member and, in its CppUnit setUp and tearDown functions, call OfficeConnection::setUp and OfficeConnection::tearDown, respectively. Tests obtain access to the OOo instance via OfficeConnection::getFactory. Similarly, a Java test class can use an org.openoffice.test.OfficeConnection instance, offering the same functionality as its C++ counterpart.

For this to work, three additional steps are necessary:

• First, in instsetoo_native, building an installation set for the openoffice product in the default language (en-US) in any package format now takes care to always also build an archive package format (tar.gz resp. zip) of that product.
• Second, building smoketestoo_native permanently installs that archive into the solver. (Except on Windows, where problems with resulting long pathnames prevent this; there, as a workaround, each test individually unzips the archive to some local temporary directory.)
• Third, a test that requires an OOo instance includes installationtest.mk in its makefile.mk. That file makes sure to call the CppUnit and JUnit test runners with the information necessary for the helper classes to start the correct OOo instance with a fresh UserInstallation and set up an URP pipe connection. The information is tunneled into the test runner executables as UNO bootstrap parameters.

unoapi Tests

The unoapi tests are redesigned to use the new test framework. They are plain JUnit tests and use the mechanisms described above to connect to running OOo instances. However, due to the large amount of logic encoded in qadevOOo, those JUnit tests for now are only thin wrappers around org.openoffice.Runner.run. This is intended to change over time.

The cwscheckapi tool to run the unoapi tests has been subsumed by subsequenttests. The latter always runs all tests, while the former had mechanisms to run only tests for modules that had been changed by a CWS. The hope is that the parallel mode of subsequenttests makes it fast enough so that running all tests all of the times becomes feasible.

Unfortunately, running the unoapi tests is still not stable enough to give useful results. Therefore, running subsequenttests has been disabled in the build bots for now. Making the tests sufficiently stable is ongoing work.

Details

Working on CWS sb118  , CWS sb120  .

CppUnit 1.12.1

• Replace existing cppunit and testshl2 combo with latest CppUnit 1.12.1.
• OOo configure --with-system-cppunit (for now only if also --with-system-stl!); minimal required version for now arbitrarily 1.12.1.
• TODO: Upstream PATCH_FILES (windows.patch?!?).
• TODO: RPATH in generated libraries and executables.
• TODO: Improve CPPUNIT_PLUGIN_EXPORT and get rid of version maps (GCC exception problems?).
• TODO: cppunit/makefile.mk MY_LIBS = -lm Solaris hack.
• TODO: In general OOo build environment, no longer set CFLAGS etc. as environment variables; have package-specific OOO_package_CFLAGS etc. and combine those into CFLAGS etc. where necessary (see OOO_STLPORT_CXXFLAGS etc. in cppunit/makefile.mk).
• TODO: Why use cppunittester instead of DllPlugInTester? (Remove building/delivering DllPlugInTester.)
• TODO: Make operator <<(std::basic_ostream &, rtl::OUString const &) from test/oustringstreaminserter.hxx generally available?
• TODO: Make STLport num_put symbols available on platforms other than Linux (see solenv/bin/addsym.awk), too, so that CPPUNIT_ASSERT_EQUAL actually outputs numeric values.

testshl2

• solenv/inc_cppunit.mk/cppunit.mk: “adapted cppunit.mk to cppunittester and no longer require ‘dmake test’ to run tests”.
• TODO: DEV300_m68 introduced TESTSHL2LIB (which I removed again) and uses it in many qa makefiles (that I need to touch someday, anyway).

Unit Tests

• OOO_SUBSEQUENT_TESTS (build.lst lists local dependencies of those directories, so that modifying a module and then calling “OOO_SUBSEQUENT_TESTS=x build” rebuilds the module and executes the tests).
• TODO: If Build Environment 2.0 would build directly to solver (instead of local output tree/deliver), that would simplify tests that currently need to take care to test local libraries instead of solver ones (see, for example, OOO_TEST_PREFIX in the services.rdb of stoc/test/uriproc/makefile.mk).
• TODO: Constructs like “-env:UNO_…=$(my_file)…” would fail for problematic characters (spaces!) in paths.

Smoke Test

• Redesign smoketestoo_native to use CppUnit and pre-installed OOo.
• TODO: For smoketestoo_native/makefile.mk “-env:arg-env=$(OOO_LIBRARY_PATH_VAR)"$${{$(OOO_LIBRARY_PATH_VAR)+=$$$(OOO_LIBRARY_PATH_VAR)}}"” mechanism: osl_executeProcess ustrEnvironments members without “=”: all platforms; documentation; documentation of osl_executeProcess_WithRedicrectedIO.
• TODO: Dynamically create smoketestdoc.sxw?
• TODO: “changed tools/debug to never fall back from writing to shell to writing to file, and read DBGSV_INIT from rtl::Bootstrap; the latter causes problems if tools/debug is called from static code executing before SAL_MAIN has called osl_setCommandArgs → removed statics from code and added assertions to osl/process; various improvements to smoketests.cxx (incl. passing an appropriate dbgsv.ini for non-pro builds)”: Instead of -env:DBGSV_INIT= in smoketest.cxx (and supporting DBGSV_INIT as bootstrap variable), globally in the build environment let DBGSV_INIT environment variable point to a dbgsv.ini with error=coredump, to catch all assertions thrown during build time (incl. ones from tests like smoke test; see Issue 109142 ).
• TODO: -headless? (appears to still display windows on Windows and Mac OS X, however, see Issue 109052 )

Pre-Installed OOo

• For Sun Hamburg Release Engineering, take MWS archive installation sets from “S:” instead of solver: “take installation set from SHIPDRIVE if appropriate”.
• TODO: PKGFORMAT=installed instead of archive?

unoapi Tests

• Known issues:
  • Issue 109517 “sc/qa/unoapi fails for ScAnnotationObj etc.”
  • Issue 109518 “sd/qa/unoapi crash in SfxViewShell::GetController”
  • Issue 109643 “toolkit/qa/unoapi tests fail when losing focus”
  • Issue 109728 “~ScAutoFormatObj uses cleared ScGlobal”
  • Issue 109770 “ChartView::getExplicitValuesForAxis accessing destroyed VCoordinateSystem”
  • Issue 109917 “fragile unoapi test using OnUnfocus event”
  • Issue 110862 “sc/qa/unoapi fails for sc.ScDataPilotTableObj::com::sun::star::sheet::XDataPilotTable::refresh()”
  • Issue 110988 “svtools/qa/unoapi fails for svtools.AccessibleBrowseBox etc.”
  • Issue 111006 “toolkit: deadlock between UnoControls”
  • Issue 111032 “sc/qa/unoapi sc.ScAccessibleCell::com::sun::star::accessibility::XAccessibleText getCharacterBounds() failure”
  • Issue 111042 “sd/qa/unoapi sd::DrawController::dispose crash”
  • Issue 111043 “sd/qa/unoapi sd::DrawController_OutlineView NullPointerException”
  • Issue 111047 “toolkit/qa/unoapi VCLXAccessibleMenuItem::doAccessibleAction deadlock”
  • Issue 111076 “vcl: [NSExtraMIData unregisterMouseEventListener:] crash”
  • Issue 111102 “chart2: ChartModelHelper::findDiagram returns null”
  • Issue 111111 “xmloff.Impress.XMLImporter::com::sun::star::document::XImporter”
  • Issue 111113 “toolkit.AccessibleStatusBarItem unoapi failure”
  • Issue 111114 “svx.AccessiblePresentationOLEShape unoapi failure”
  • Issue 111148 “forms/qa/unoapi forms.OImageControlModel::com::sun::star::beans::XMultiPropertySet deadlock”
• Missing tests:
  • Issue 109564 “qadevOOo/tests/java/ifc/awt/_XSystemChildFactory.java”
• Sporadic failures:
  • Issue 100159 “unexpected exception from oooimprovement during shut down”
  • Issue 109939 “toolkit: inconsistent UnoControl locking schemes”
  • Issue 110253 “chart2: chart::WrappedPropertySet::setPropertyValue crash”