Revision as of 09:34, 16 September 2009

The taming of the shrew

A current build contains roughly 200 "modules", top level directories in the repository. Our build system tracks build dependencies between them in the "build.lst" files in the "prj" directories of them. Not every tracked dependency is necessary or useful, some dependencies have been forgotten, some were never correct, some can be avoided by a few code or makefile changes. This is tracked in issue 103496.

If all dependencies are drawn in a module graph, the result Media:Complete.odg is hilarious and chaotic, due to the fact that at both ends of the graph some modules collect a lot of dependencies. At the lower end it's solenv as this is needed always and at the upper end it's postprocess as it "depends" on every module that without postprocess would be a top level module. The module instsetoo_native and smoketestoo_native depend on postprocess. By removing these modules as well as dmake and guw (as they are built outside the ooo build process) the graph Media:reduced.odg looks a bit better, but it's still too much as especially the indirect dependencies make it impossible to see the forest for the trees.

It should be possible to group modules, either by their "level" in our project or because they are somewhat related. Some of the mentioned unnecessary or bad dependencies prevent a proper separation between the groups, so fixing that by moving or changing source code is necessary. Here's a suggested group list, based on the DEV300_m57 build and the fixes from issue 103496 applied. I have created the groups so that their modules basically should have dependencies only on each other or to modules from a "lower" group. In some cases there are some dependencies that violate that principle (I will mention them below), but I'm confident that they can be fixed by changing code or module content. Issue 103496 or any follow-up issues will track that progress.

Build prerequisites

Some modules are needed before even the first "real" module can be built or they can be seen as part of the build environment: solenv, soltools, postprocess, instsetoo_native, dmake, guw. soltools still has a "dirty" dependency on stlport (that itself is part of the external modules), but must be built before any other module than stlport. This must be solved by either removing the stl usage, using the compiler stl in soltools or in the complete office. The last variant would make our UNO C++ language binding incompatible, so it needed some coordination. solenv mainly consists of scripts and makefile includes.

External modules

These comprises the following modules: afms, agg, apache-commons, apple_remote, beanshell, berkeleydb, bitstream_vera_fonts, boost, cairo, cppunit, curl, epm, expat, external, fondu, graphite, hsqldb, hunspell, hyphen, icc, icu, jfreereport, jpeg, libegg, libtextcat, libwpd, libxml2, libxmlsec, libxslt, lpsolve, lucene, MathMLDTD, moz, msfontextract, neon, np_sdk, nss, openssl, python, redland, rhino, sane, saxon, stax, stlport, tomcat, twain, unixODBC, vigra, x11_extensions, xpdf, xsltml, zlib.

With a few exceptions all code modules are checked in as download + patch, "external" contains binary stuff that is not part of the repository. This group is "clean", means: it does not have any dependency on modules of any of the following groups.

URE - UNO Runtime Environment

In former times we even have built this as a separate product, but nowadays it's only a part of the ODK (OpenOffice.org Development Kit). All parts of the URE only depend on each other or on some external modules (so that a packaged URE could need some of them, e.g. stlport). A packaged URE would need a license file that currently would create a build dependency on readlicense_oo. Though no localized license files would be needed, the fact that readlicense_oo also does the localization would create a dependency on the localization tooling. An english build runs fine even without that (and packaging would be done in instsetoo_native anyway), so I have omitted that for the moment and so readlicense_oo is not a part of the URE list:

bridges, cli_ure, codemaker, cppu, cppuhelper, cpputools, idlc, io, javaunohelper, jurt, jvmaccess, jvmfwk, offapi, offuh, pyuno, rdbmaker, registry, remotebridges, ridljar, sal, salhelper, stoc, store, testshl2, udkapi, unoil, ure, xml2cmp.

I have added pyuno though it never was or currently also not is part of the URE, but it feels like being a part of it.

In my current split build based on these groups I have to add the "testshl2" module here also, though it is not related to the URE. But the testshl2 module is needed to execute C++ unit tests and if we want to run them while building, we need to build this module as early as possible. It's very unfortunate that our C++ tests don't use "pure" CppUnit but unstead added some code that depends on sal. Another bad side effect of this is that the sal C++ tests currently can't be executed in the build. This should be fixed by removing testshl2 and using CppUnit directly.

The list contains offuh and offapi as currently the only way to generate UNO API headers is using offuh. As soon as we replace offuh by building headers inside of offapi and udkapi, we can remove offuh and offapi. The former would just go away, while the latter would become a part of the next group:

ODK - OpenOffice.org Development Kit

This group comprises bean, odk, autodoc, udm, cosv, sandbox, unodevtools. In the "odk" module all parts of an ODK installation set are collected in some zip files that will be used to create an installation set in instsetoo_native. In future this group also may contain offapi, as explained in the URE section. Autodoc and unodevtools are tools needed by the ODK or its users (udm and cosv are prerequisites of autodoc). "bean" is an SDK example and uses the "sandbox" module that until now also was needed by the office module "sj2". But as this module will be removed in one of the next milestones, I already moved it here.

The ODK currently uses GNU make (and some tools from the URE) for builds, while the OOo build system uses build.pl and dmake. It would be nice to have both systems united, so that not only all extensions but also OOo could be built with an ODK. The future will show how and how fast we can achieve that.

The ODK group does not need to be built for an OOo build - nothing here is used to build or run OOo.

Build and test tools

Beside solenv, dmake, soltools, cppunit/testshl2 and the URE/ODK build tools some other tools are needed to build and test OOo. This comprises the following modules:

building: idl, rsc

They should depend only on external or URE modules. Currently that isn't true, mainly due to a dependency on "tools". So currently in a real build these modules must stay in the big "office" group.

testing: automation, smoketestoo_native, testautomation, testtools, qadevooo

"automation" is needed to use the OOo testtool for GUI tests that themselves are found in "testautomation". "smoketestoo_native" is the first test after successful creation of an OOo installation set that can be executed in the build process. "qadevooo" is used by API and complex tests. Some of them are executed while the build is running, so this module needs to be built together with the large "office" group. "testtools" contains some other test tools that currently are not used inside of an OOo build.

Some special modules

Before we come to the "real thing", here are some modules that don't fit well into any other categorie. Some of them deliver parts of an OOo installation set:

crashrep, extras, helpcontent2, l10n, readlicense_oo, binfilter, hwpfilter, javainstaller2

while some don't:

migrationanalysis, soldep

I've put them aside for now. IMHO all of them except crashrep and readlicense_oo are good candidates for separate source code repositories. binfilter is added here as it is no real extension, but also isn't part of the office core. hwpfilter also isn't an extension but could be easily converted into it.

Unfortunately the l10n module (and so the transex3 tool) is needed even to create an english build. This has to be investigated.

readlicense_oo is only needed for packaging, but not for building, with the exception of swext and sdext. Other extensions (dictionaries, reportbuilder have their own license files).

Images

All images in OOo will be packed into images.zip. This requires the modules

default_images, external_images, ooo_custom_images, packimages

With the exception of packimages no other modules should get images directly from the images modules. Unfortunately the images can't be built and used separately as the rsc compiler checks images while it is used in the office modules and packimages needs the image list resources to know which images should be packed. So the image modules need to be present while OOo is building and the packimages step must be done after everything else has been built.

Extensions

Some modules are built as extensions:

sdext, swext, dictionaries, reportbuilder

Building extensions should not require anything than an ODK. But most of them still contain some "bad" dependencies:

• sdext (or more specifically: pdfimport) uses comphelper inlines and links statically against basegfx
• reportbuilder depends on wizards
• dictionaries depends on readlicense_oo

It would be nice if "binfilter" could be added here, but unfortunately it still has dependencies on office libraries that can't be removed easily.

Office modules

Everything else can be seen as an "Office" module. It can be helpful to differentiate between "top level" modules (those who don't have others depending on them) and "basis" modules. Some modules are "real" UNO components have only very low level dependencies (URE and external) or are close to that (only dependencies on e.g. tools and comphelper), others have dependencies on the usual suspects (vcl, svx ...).

Top Level modules

accessibility, basctl, chart2, configmgr, desktop, dtrans, embeddedobj, embedserv, eventattacher, extensions, filter, forms, fpicker, hwpfilter, lingucomponent, package, padmin, psprint_config, reportdesign, sc, scaddins, sccomp, scp2, scripting, sd, slideshow, starmath, sw, ucb, UnoControls, unoxml, uui, wizards, xmerge, xmlsecurity

Basis Office modules

animations, avmedia, basebmp, basegfx, basic, canvas, comphelper, connectivity, cppcanvas, dbaccess, drawinglayer, fileaccess, formula, framework, goodies, i18npool, i18nutil, linguistic, o3tl, officecfg, oovbaapi, oox, regexp, sax, setup_native, sfx2, shell, sj2, sot, svtools, svx, sysui, toolkit, tools, ucbhelper, unotools, uui, vcl, vos, writerfilter, writerperfect, xmlhelp, xmloff, xmlscript

The goal for these modules should be to combine some of them to few and larger ones (and rearranging the code in them).