Test Cleanup

From Apache OpenOffice Wiki
Revision as of 10:53, 24 January 2011 by Sb (talk | contribs) (→‎unoapi Tests)
Jump to: navigation, search

Test Cleanup Project

The demand for automated (unit) testing should be obvious. See the mission statement mail and slides from a FOSDEM'10 talk and an OOoCon 2010 presentation for more information and further links.

For examples of how to write xUnit based tests in the OOo code base, see basegfx/test/, sal/qa/ByteSequence/, and smoketest_native/. However, the hard part should not be how to write test code, anyway, but how to write testable code; see The Clean Code Talks—Unit Testing.

When testing C++ code that merely uses UNO types (no services), but uses them in a way that requires UNO to be bootstrapped so that the relevant type information is available (e.g., putting instances of complex UNO types into com::sun::star::uno::Any), one approach is as follows: In the makefile.mk, instead of .INCLUDE: _cppunit.mk use 

.IF "$(OS)" == "WNT"
my_file = file:///
.ELSE
my_file = file://
.END

ALLTAR : test

test .PHONY : $(SHL1TARGETN)
    $(CPPUNITTESTER) $(SHL1TARGETN) \
        -env:UNO_TYPES=$(my_file)$(SOLARBINDIR)/types.rdb

and in the test code once call cppu::defaultBootstrap_InitialComponentContext().

User Interface

CppUnit

For tests in C++, plain CppUnit (at least version 1.12.1) is available. (OOo bundles CppUnit 1.12.1 as an external module; alternatively, you can configure --with-system-cppunit.)

Implement the test code as a dynamic library, following CppUnit conventions (i.e., using CPPUNIT_TEST_SUITE_REGISTRATION and CPPUNIT_PLUGIN_IMPLEMENT macros). The following settings are needed in the makefile.mk:

  • ENABLE_EXCEPTIONS = TRUE
  • CFLAGSCXX += $(CPPUNIT_CFLAGS)
  • SHLnRPATH = NONE
  • SHLnSTDLIBS += $(CPPUNITLIB)
  • SHLnVERSIONMAP = version.map, where that file must contain the following (see details below for improving this):
 UDK_3_0_0 {
   global:
     cppunitTestPlugIn;
   local:
     *;
 };

To run tests, sal/cppunittester delivers a cppunittester executable that has some advantages over the plain CppUnit DllPlugInTester:

  • It uses SAL_IMPLEMENT_MAIN to properly set up the per-process settings required by URE. Test code as well as tested code based on URE will expect this and would fail in a process that is not set up in this way.
  • Getting CppUnit to build as an external module within the Windows OOo build is a non-trivial task. The less parts of it that need to be built, the better.

Something like the following in makefile.mk will execute the tests: 

 ALLTAR : test
 test .PHONY : $(SHL1TARGETN)
   $(CPPUNITTESTER) $(SHL1TARGETN)

$(CPPUNITTESTER) internally uses $(AUGMENT_LIBRARY_PATH_LOCAL) to favor library instances in the local output tree over ones in solver. Alternatively to the above, .INCLUDE: _cppunit.mk calls $(CPPUNITTESTER) on each of $(SHL1TARGETN), …, $(SHL10TARGETN) that is defined.

JUnit

For tests in Java, plain JUnit (at least version 4) is available. (OOo requires JUnit as a prerequisite, see junit.org for downloads; you can configure --with-junit=… if it is not automatically found; alternatively, you can configure --without-junit, disabling JUnit-based tests.)

subsequenttests

Some tests have build and/or run time dependencies on modules further up in the hierarchy, or modules that the rest of the module in question does not have dependencies on (and which are thus not listed in the module's build.lst). The extreme case are tests that need a complete OOo installation as their test environment.

For such tests, the whole makefile.mk should be wrapped in: 

 .IF "$(OOO_SUBSEQUENT_TESTS)" == ""
 nothing .PHONY :
 .ELSE
 ...
 .END

(and the directory included in the module's build.lst).

The idea is to do a complete OOo build first (cd smoketestoo_native && build --all) and then build and execute those tests either selectively or wholesale.

Selectively building a specific such test directory can be done with cd … && OOO_SUBSEQUENT_TESTS=TRUE dmake.

To build and execute all these test directories, the tool subsequenttests (in solenv/bin) exists. It scans the build.lst files of all modules for makefile.mks that mention OOO_SUBSEQUENT_TESTS and simply executes dmake in all those directories, with OOO_SUBSEQUENT_TESTS set. It supports a -Pn switch to build n directories in parallel.

Helpers

A new module test shall collect reusable code snippets for C++ and Java tests.

For now, it concentrates on letting a test connect to an OOo instance via remote UNO: A C++ test class can hold a test::OfficeConnection as class member and, in its CppUnit setUp and tearDown functions, call OfficeConnection::setUp and OfficeConnection::tearDown, respectively. Tests obtain access to the OOo instance via OfficeConnection::getFactory. Similarly, a Java test class can use an org.openoffice.test.OfficeConnection instance, offering the same functionality as its C++ counterpart.

For this to work, three additional steps are necessary:

  • First, in instsetoo_native, building an installation set for the openoffice product in the default language (en-US) in any package format now takes care to always also build an archive package format (tar.gz resp. zip) of that product.
  • Second, building smoketestoo_native permanently installs that archive into the solver. (Except on Windows, where problems with resulting long pathnames prevent this; there, as a workaround, each test individually unzips the archive to some local temporary directory.)
  • Third, a test that requires an OOo instance includes installationtest.mk in its makefile.mk. That file makes sure to call the CppUnit and JUnit test runners with the information necessary for the helper classes to start the correct OOo instance with a fresh UserInstallation and set up an URP pipe connection. The information is tunneled into the test runner executables as UNO bootstrap parameters.

For debugging purposes, which OOo instance the tests use can manually be overridden, see “OOO_TEST_SOFFICE: allow override of tested soffice process”.

unoapi Tests

The unoapi tests are redesigned to use the new test framework. They are plain JUnit tests and use the mechanisms described above to connect to running OOo instances. However, due to the large amount of logic encoded in qadevOOo, those JUnit tests for now are only thin wrappers around org.openoffice.Runner.run. This is intended to change over time.

The cwscheckapi tool to run the unoapi tests has been subsumed by subsequenttests. The latter always runs all tests, while the former had mechanisms to run only tests for modules that had been changed by a CWS. The hope is that the parallel mode of subsequenttests makes it fast enough so that running all tests all of the times becomes feasible.

Unfortunately, running the unoapi tests is still not stable enough to give useful results. Therefore, running subsequenttests has been disabled in the build bots for now. Making the tests sufficiently stable is ongoing work.

Details

Working on CWS sb118  , CWS sb120  .

CppUnit 1.12.1

  • Replace existing cppunit and testshl2 combo with latest CppUnit 1.12.1.
  • OOo configure --with-system-cppunit (for now only if also --with-system-stl!); minimal required version for now arbitrarily 1.12.1.
  • TODO: Upstream PATCH_FILES (windows.patch?!?).
  • TODO: RPATH in generated libraries and executables.
  • TODO: Improve CPPUNIT_PLUGIN_EXPORT and get rid of version maps (GCC exception problems?).
  • TODO: cppunit/makefile.mk MY_LIBS = -lm Solaris hack.
  • TODO: In general OOo build environment, no longer set CFLAGS etc. as environment variables; have package-specific OOO_package_CFLAGS etc. and combine those into CFLAGS etc. where necessary (see OOO_STLPORT_CXXFLAGS etc. in cppunit/makefile.mk).
  • TODO: Why use cppunittester instead of DllPlugInTester? (Remove building/delivering DllPlugInTester.)
  • TODO: Make operator <<(std::basic_ostream &, rtl::OUString const &) from test/oustringstreaminserter.hxx generally available?
  • TODO: Make STLport num_put symbols available on platforms other than Linux (see solenv/bin/addsym.awk), too, so that CPPUNIT_ASSERT_EQUAL actually outputs numeric values.

testshl2

Unit Tests

  • OOO_SUBSEQUENT_TESTS (build.lst lists local dependencies of those directories, so that modifying a module and then calling “OOO_SUBSEQUENT_TESTS=x build” rebuilds the module and executes the tests).
  • TODO: If Build Environment 2.0 would build directly to solver (instead of local output tree/deliver), that would simplify tests that currently need to take care to test local libraries instead of solver ones (see, for example, OOO_TEST_PREFIX in the services.rdb of stoc/test/uriproc/makefile.mk).

Template:Documentation/Note

  • TODO: Constructs like “-env:UNO_…=$(my_file)…” would fail for problematic characters (spaces!) in paths.

Smoke Test

Pre-Installed OOo

unoapi Tests

  • Known issues, open (not yet addressed at all; disabled for now on net-yet-integrated CWS; or fixed on not-yet-integrated CWS):
    • Issue 116120 “vcl: TabControl::ImplGetTabRect negative height leads to X_CreatePixmap BadAlloc”
    • Issue 116125 “qadevOOo/qa/complex/junitskeleton failure”
    • Issue 116156 “toolkit/qa/unoapi crash in SwAccessibleMap::RemoveContext”
    • Issue 116192 “[sw] fixed text fields sometimes get updated”
    • Issue 116550 “toolkit/qa/unoapi crashes OOo in ~SwAccessibleParagraph vs. SwXTextDocument::close”
    • Issue 116583 “comphelper/qa/complex/comphelper failure”
  • Known issues, disabled for now:
    • Issue 109517 “sc/qa/unoapi fails for ScAnnotationObj etc.”
    • Issue 109518 “sd/qa/unoapi crash in SfxViewShell::GetController”
    • Issue 109643 “toolkit/qa/unoapi tests fail when losing focus”
    • Issue 109728 “~ScAutoFormatObj uses cleared ScGlobal”
    • Issue 109917 “fragile unoapi test using OnUnfocus event”
    • Issue 109939 “toolkit: inconsistent UnoControl locking schemes”
    • Issue 110862 “sc/qa/unoapi fails for sc.ScDataPilotTableObj::com::sun::star::sheet::XDataPilotTable::refresh()”
    • Issue 110988 “svtools/qa/unoapi fails for svtools.AccessibleBrowseBox etc.”
    • Issue 111006 “toolkit: deadlock between UnoControls”
    • Issue 111032 “sc/qa/unoapi sc.ScAccessibleCell::com::sun::star::accessibility::XAccessibleText getCharacterBounds() failure”
    • Issue 111042 “sd/qa/unoapi sd::DrawController::dispose crash”
    • Issue 111043 “sd/qa/unoapi sd::DrawController_OutlineView NullPointerException”
    • Issue 111102 “chart2: ChartModelHelper::findDiagram returns null”
    • Issue 111111 “xmloff.Impress.XMLImporter::com::sun::star::document::XImporter”
    • Issue 111113 “toolkit.AccessibleStatusBarItem unoapi failure”
    • Issue 111148 “forms/qa/unoapi forms.OImageControlModel::com::sun::star::beans::XMultiPropertySet deadlock”
    • Issue 111169 “svx.AccessiblePageShape unoapi failure”
    • Issue 111170 “sun/i18n/qa/unoapi "creating a drawdoc" failure”
    • Issue 111178 “sw/qa/unoapi SwView::SelectShell crash”
    • Issue 111180 “framework/qa/unoapi fwk.Desktop::com::sun::star::frame::XComponentLoader::loadComponentFromURL() failure”
    • Issue 111182 “Desktop::terminate -> ~DropTarget -> _objc_error crash”
    • Issue 111184 “sal/qa/osl/pipe uses fixed pipe names”
    • Issue 111186 “sw/qa/unoapi sw.PageStyle::com::sun::star::beans::XPropertySet::setPropertyValue() failure”
    • Issue 111190 “sw/qa/unoapi sw.SwXShape failure”
    • Issue 111194 “sw/qa/unoapi sw.XMLExporter crash”
    • Issue 111195 “toolkit/qa/unoapi toolkit.AccessibleScrollBar::com::sun::star::accessibility::XAccessibleValue::setCurrentValue() failure”
    • Issue 111197 “sw/qa/unoapi sw.SwAccessibleDocumentPageView failure”
    • Issue 111199 “sd/qa/unoapi sd.AccessibleOutlineView failure”
    • Issue 111200 “xmloff/qa/unoapi xmloff.Draw.XMLMetaImporter failure”
    • Issue 111216 “svx/qa/unoapi accessibility deadlock”
    • Issue 111218 “sw/qa/unoapi sw.SwAccessibleParagraphView::com::sun::star::accessibility::XAccessibleEventBroadcaster::removeEventListener() failure”
    • Issue 111220 “sw/qa/unoapi sw.XMLContentExporter::com::sun::star::document::XFilter::cancel() failure”
    • Issue 111224 “xmloff/qa/unoapi xmloff.Impress.XMLContentExporter crash”
    • Issue 111225 “toolkit/qa/unoapi toolkit.AccessibleStatusBar failure”
    • Issue 111252 “svx/qa/unoapi svx.AccessibleOLEShape deadlock”
    • Issue 111273 “sw/qa/unoapi sw.SwXTextEmbeddedObject::com::sun::star::document::XEmbeddedObjectSupplier::getEmbeddedObject() crash”
    • Issue 111278 “svx/qa/unoapi svx.AccessibleEditableTextPara failure”
    • Issue 111283 “sfx2/qa/unoapi sfx.StandaloneDocumentInfo failure”
    • Issue 111285 “forms/qa/unoapi forms.OButtonModel failure”
    • Issue 111287 “xmloff/qa/unoapi XMLSettingsIm-/Exporter crash”
    • Issue 111329 “sd/qa/unoapi SdXCustomPresentationAccess::getElementNames/ImpEditEngine::OnlineSpellHdl hang”
    • Issue 111332 “sw/qa/unoapi sw.XMLStylesExporter::com::sun::star::document::XFilter crash”
    • Issue 111333 “forms/qa/unoapi forms.OImageControlControl::com::sun::star::awt::XControl deadlock”
    • Issue 112743 “starmath/qa/unoapi sm.XMLSettingsExporter failure”
    • Issue 112746 “framework/qa/unoapi fwk.ModuleManager failure (X11 BadWindow)”
    • Issue 112751 “testgraphical/qa/graphical failure”
    • Issue 112778 “xmloff/qa/unoapi xmloff.Draw.XMLContentExporter failure”
    • Issue 113020 “sw/qa/unoapi sw.SwXTextGraphicObject::com::sun::star::beans::XPropertySet failure”
    • Issue 113103 “chart2: deadlock between DiagramHelper and Diagram”
    • Issue 113136 “sw/qa/unoapi sw.SwAccessibleParagraphView::com::sun::star::accessibility::XAccessibleEditableText failure”
    • Issue 113142 “sw/qa/unoapi SwAccessibleContext::RemoveFrmFromAccessibleMap crash”
    • Issue 113161 “sd/qa/unoapi sd.SdLayer::com::sun::star::drawing::Layer::IsVisible failure”
    • Issue 113201 “forms/qa/unoapi forms.OEditModel::com::sun::star::form::validation::XValidatableFormComponent failure”
    • Issue 113245 “framework/qa/unoapi fwl.TypeDetection failure”
    • Issue 113306 “sfx2/qa/unoapi sfx.DocumentTemplates failure”
    • Issue 113386 “sd/qa/unoapi sd.AccessibleDrawDocumentView::com::sun::star::accessibility::XAccessibleComponent::containsPoint() failure”
    • Issue 113489 “toolkit/qa/unoapi toolkit.AccessibleMenu{,Bar,Separator} failures”
    • Issue 114205 “sw/qa/unoapi sw.SwAccessibleParagraphView::com::sun::star::accessibility::XAccessibleText::copyText() failed”
    • Issue 114209 “forms/qa/unoapi forms.OGridControlModel::com::sun::star::view::XSelectionSupplier crash”
    • Issue 114211 “xmloff/qa/unoapi xmloff.Draw.XMLStylesImporter::com::sun::star::lang::XInitialization crash”
    • Issue 114213 “toolkit/qa/unoapi toolkit.AccessibleMenu::com::sun::star::accessibility::XAccessibleSelection failure”
    • Issue 114636 “toolkit/qa/unoapi toolkit.AccessibleScrollBar::com::sun::star::accessibility::XAccessibleAction failure”
    • Issue 114637 “sw/qa/unoapi sw.SwAccessibleFootnoteView(?) crash”
    • Issue 114642 “svx/qa/unoapi svx: svx.SvxUnoTextContent::com::sun::star::style::CharacterProperties crash”
    • Issue 114669 “forms/qa/unoapi forms.OListBoxControl crash”
    • Issue 114889 “sw/qa/unoapi sw.SwXTextViewCursor::com::sun::star::view::XScreenCursor”
  • Missing tests:
    • Issue 109564 “qadevOOo/tests/java/ifc/awt/_XSystemChildFactory.java”
  • Sporadic failures:
    • Issue 111217 “BasicManager::SetGlobalUNOConstant crash during shutdown”
    • Issue 111246 “svx::FontHeightToolBoxControl::statusChanged crash (memory corruption)”
    • Issue 111271 “Mac OS X: crash during shutdown, __tcf_2 on stack”
    • Issue 111400 “Extension Manager does not join spawned uno process (-> smoketestoo_native fails)”
    • Issue 112070 “UNO API tests must not use random”
    • Issue 113158 “crash with two threads in framework::ToolBarManager dtor”
    • Issue 113197 “sw/qa/unoapi sw.SwXTextRange hangs”
    • Issue 113367 “sysdtrans deadlock during shutdown”
    • Issue 113464 “stoc_smgr::ORegistryServiceManager::dispose crash during shutdown”
    • Issue 114207 “framework::ModuleUIConfigurationManager::dispose crash during shutdown”
    • Issue 114675 “binary URP bridge race on shut down, crash”
    • Issue 114718 “forms/qa/unoapi: crash in remote async release call”
    • Issue 116131 “vcl: message unregisterDraggingDestinationHandler: sent to freed object”
Retrieved from "https://wiki.openoffice.org/w/index.php?title=Test_Cleanup&oldid=192348"
Personal tools