Revision as of 14:41, 25 October 2010

Motivation

UNO components in the various language bindings (dynamic libraries for binary/C++ UNO, jars for Java UNO, etc.) have mechanisms to actively report information about the services and singletons they offer (exported symbol component_writeInfo for dynamic libraries, public static __writeRegistryServiceInfo method for jars, etc.). This active component registration has various drawbacks:

• Calls to actively register the components (regcomp) need to be made in carefully crafted environments, ensuring that the relevant UNO components can actually be loaded and their code executed. Especially for Java and Python components, this is nontrivial.
  • At OOo build time, when a services.rdb is assembled in instsetoo_native.
  • At OOo installation time, when bundled extensions that contain UNO components are registered.
  • At OOo run time, when extensions that contain UNO components are registered or revoked.
• Loading the UNO components to actively execute their code can be time consuming (especially if the components' functionality is not otherwise needed, like when assembling a services.rdb at build time).

Therefore, it is sought to replace active component registration with a passive approach, where a UNO component is accompanied by a representation of the relevant information that can be obtained without running the component's code.

An XML Format

The canonic choice for representing such information these days is XML. The necessary information is encoded as follows:

An XML document encoding information about multiple UNO components (i.e., corresponding to a services.rdb) has a root element with local name components and namespace URI http://openoffice.org/2010/uno-components.

The components root element contains zero or more elements with local name component and namespace URI http://openoffice.org/2010/uno-components, each with an attribute with unprefixed name uri, the value of which is a URI (in OOo-internal form) denoting the relevant UNO component (it may be a vnd.sun.star.expand URL; it may be a relative URL, considered relative to the URL of the XML document itself), and an attribute with unprefixed name loader, the value of which is the name of the relevant UNO service with which to load the component (e.g., com.sun.star.loader.SharedLibrary or com.sun.star.loader.Java2). Each component element in turn contains zero or more service and/or singleton elements.

Each element with local name service or singleton and namespace URI http://openoffice.org/2010/uno-components has an attribute with unprefixed name name, the value of which is the name of the respective service or singleton.

(The information provided by active component registration contains redundancies that are dropped from this XML format. For one, UNO components are listed in an IMPLEMENTATIONS section and each service and singleton is again listed in a SERVICES resp. SINGLETONS section. For another, each singleton specifies the service implementing it. For the latter to work, the textualservices Key implementation throws an exception when asked for the associated service, and cppu::bootstrapInitialContext had to be adapted to use the REGISTERED_BY information instead.)

There was already a somewhat similar textual format for regcomplazy in use, which however was based on ini-file syntax (but with irregularities, containing lines that do not match the key=value format) instead of XML. This has been superseded by the approach described here, and regcomplazy has been removed.

As we will see later, within the OOo build system there are also XML files describing single UNO components, and which have a component element instead of a components element as their root element, but otherwise use the same structure as the above XML format.

XML at Run Time

There was a choice whether to either take the XML files and inject them into registry-format services.rdb files (i.e., to still call regcomp resp. unopkg at build resp. installation time), or to use the XML files directly at run time. I decided for the latter.

The configmgr re-write showed that it is acceptable performance-wise to read a handful of XML files during OOo start up, using an XML reader specifically written for that task. So it should be acceptable to replace the existing registry-format services.rdb files (for a start, the URE layer services.rdb and the basis layer services.rdb and legacy_binfilters.rdb) with XML files.

The advantages are easier and potentially faster creation (no need for regcomp like code any more), and easier manipulation (due to the textual format). Also, the registry format has a known design defect, in that it cannot handle arbitrary combinations of key names.

Work in Progress

Work is done on CWS sb129   and CWS sb131   in the following logical steps:

• “simpleregistry-cleanup”: f02f9ae6b514 cleans up the stoc/source/simpleregistry code so that subsequent patches can better modify it.

• “xmlreader”: 18115c689f48 and 3c20685784c2 extract the XmlReader code from configmgr to a new xmlreader URE module, so that it can be reused from stoc. An open issue is to version its C++ ABI if it ever evolves, see Issue 115203 .

• “textualservices”: c14649ec0dfd, 1f180e2c69b6, ebf1e66a64f8, and 4d85756a1af9 extend the implementation of com.sun.star.registry.SimpleRegistry in stoc/source/simpleregistry, so that it can internally also read XML files in addition to registry files, and present them to its clients as if they had the structure of registry-format services.rdb files. This appeared to be the simplest approach for now, given how deep the knowledge about registry-format files is rooted in the code base (see, for example, the functionality in cppuhelper/bootstrap.hxx that bootstraps a UNO environment based on rdb files); it is still subject to change, however.

• “passive”: 86c20aebcd79, 16b19de9f006, c4897aa8183f, 3c117e628229, c17dd319ab10, and 812a907cb009 replace the active registration of all of OOo's UNO components with passive registration. The resulting XML files are still called services.rdb etc. (instead of, say, services.xml), mainly because [ure/source/README] declares the URE services.rdb as part of the published URE interface (acknowledging only its existence, not its internal format, and especially warning against registering anything into it).

• “tests”: 55b03b1b8257 adapts OOO_SUBSEQUENT_TESTS-based tests (that used to set up their necessary environments via regcomp). (Also, subsequenttests is slightly improved and now supports make-like switches -k and -n.) Tests want to prefer data in the local module output tree over the corresponding, delivered data in the solver, which complicates things. For the tests adapted here, this is solved via calling cppu::defaultBootstrap_InitialComponentContext with explicitly set UNO_TYPES and UNO_SERVICES, plus as follows:
  • cppuhelper/qa/propertysetmixin: cppu::defaultBootstrap_InitialComponentContext is modified to look for the bootstrap.uno dynamic library under bootstrap variable URE_INTERNAL_LIB_DIR if that is set, and only next to the cppuhelper library otherwise. The cppuhelper dynamic library is itself no UNO component, so it works to set URE_INTERNAL_LIB_DIR to solver. The test’s internal components are accessed via COMPONENTPREFIX_INBUILD_NATIVE/JAVA.
  • stoc/test/uriproc: The test only instantiates services from the stocservices.uno dynamic library, so it works to set URE_INTERNAL_LIB_DIR to the module’s local output tree (using a locally created services.rdb, in case there are changes to stocservices.component that have not yet been reflected in the solver’s ure/services.rdb).

• Issue 114962 “passive registration of UNO components in extensions”, documented at Documentation/DevGuide/Extensions/File_Format#Passively_Registered_UNO_Components.

Adapting the OOo SDK, the NetBeans OOo Plugin, etc. is addressed by Issue 114609 .

.component Files

For each UNO component implementation in the OOo code base that does not implement a UNO component that is included in backwards-compatible extensions (see below), any existing active registration code is removed, and instead a X.component file is added. That file describes component X in the above XML format, with a component root element that lacks the uri attribute (because it is platform dependent, e.g., libvclli.so vs. vclmi.dll). Then, a makefile.mk rule uses new solenv/bin/createcomponent.xslt to turn this into a proper .component file including uri. (Remember that using $(XSLTPROC) implies a module dependency on LIBXSLT:libxslt.) New postprocess/packcomponents/makefile.mk uses new solenv/bin/packcomponents.xslt to bundle those .component files into the basis-layer services.rdb (similarly, URE-layer services.rdb is assembled in ure/source/makefile.mk and basis-layer legacy_binfilters.rdb is assembled in binfilter/util/makefile.mk).

The makefile.mk rules calling createcomponent.xslt use new settings.mk variables denoting the paths to the components:

• COMPONENTPREFIX_URE_NATIVE: A dynamic library component in the URE layer.
• COMPONENTPREFIX_URE_JAVA: A jar component in the URE layer.
• COMPONENTPREFIX_BASIS_NATIVE: A dynamic library component in the basis layer.
• COMPONENTPREFIX_BASIS_JAVA: A jar component in the basis layer.
• COMPONENTPREFIX_BASIS_PYTHON: A Python component in the basis layer.
• COMPONENTPREFIX_INBUILD_NATIVE: A dynamic library component used during the build; a process using it must set the UNO bootstrap variable OOO_INBUILD_SHAREDLIB_DIR accordingly.
• COMPONENTPREFIX_INBUILD_JAVA: A jar component used during the build; a process using it must set the UNO bootstrap variable OOO_INBUILD_JAR_DIR accordingly.
• COMPONENTPREFIX_EXTENSION: A component (dynamic library, jar, Python) bundled in an OOo extension.

The corresponding information about UNO component registration was removed from the scp2 data. One problem is that what components were actually registered into a registry-format services.rdb in instsetoo_native depended on what scp2 File definitions are visible when building a product. This varies among products (especially between OpenOffice.org and Oracle Open Office), so that different products used to silently produce different basis-layer services.rdb files (even though this must not be). With the new, more static approach of assembling services.rdb files, this needs to be addressed properly (see TODOs in postprocess/packcomponents/makefile.mk).

The below throwaway Haskell script was used to translate the regview output of an existing registry-format services.rdb into XML format, copying the resulting component elements into the individual .component files:

-- Makefile:
--  .PHONY: components
--  components:
--   ghc --make -o $@ components.hs
--
-- Extract components XML data from regview output:
--
-- $ regview ... | components > ...
 
{-# LANGUAGE ScopedTypeVariables #-}
 
module Main (main) where
 
import Data.Map (Map, alter, empty, foldWithKey, insert, singleton, toList)
import Data.List (sort)
import Text.Regex.Posix ((=~))
 
type Impls = Map String Impl -- keyed on implementation name
data Impl =
    Impl { activator :: String, location :: String, services :: [String],
           singletons :: [String] }
    deriving Show
 
type Comps = Map String Comp -- keyed on URI
data Comp = Comp { loader :: String, impls :: Impls }
    deriving Show
 
data State = StateStart | StateImplementations | StateImplementation String
           | StateActivator String | StateLocation String | StateServices String
           | StateSingletons String
 
(=~~~) :: String -> String -> Maybe [String]
a =~~~ b = if c /= "" then Just ds else Nothing
    where (_::String, c::String, _::String, ds) = a =~ b
 
getImpl :: Maybe Impl -> Impl
getImpl (Just impl) = impl
getImpl Nothing = Impl undefined undefined [] []
 
setActivator :: Impls -> String -> String -> Impls
setActivator c id n = alter alt id c
    where alt x = let i = getImpl x in Just $ i { activator = n }
 
setLocation :: Impls -> String -> String -> Impls
setLocation c id n = alter alt id c
    where alt x = let i = getImpl x in Just $ i { location = n }
 
addService :: Impls -> String -> String -> Impls
addService c id n = alter alt id c
    where alt x = let i = getImpl x in Just $ i { services = n : services i }
 
addSingleton :: Impls -> String -> String -> Impls
addSingleton c id n = alter alt id c
    where alt x = let i = getImpl x
                  in Just $ i { singletons = n : singletons i }
 
parseLine :: [State] -> Impls -> IO ([State], Impls)
parseLine s c = do l <- getLine
                   return $ parseLine' s l
    where
      parseLine' ss@(_ : sr) l =
          case parseLine'' ss l of
            Just a -> a
            _ -> parseLine' sr l
      parseLine' [] l = (s, c)
      parseLine'' ss@(StateStart : _) l =
          if l == " / IMPLEMENTATIONS" then Just (StateImplementations : ss, c)
          else Nothing
      parseLine'' ss@(StateImplementations : _) l =
          case l =~~~ "^   / (.+)$" of
            Just [id] -> Just (StateImplementation id : ss, c)
            _ -> case l =~~~ "^ / " of
                   Just _ -> Just ([], c)
                   _ -> Nothing
      parseLine'' ss@(StateImplementation id : _) l =
          if l == "       / ACTIVATOR" then Just (StateActivator id : ss, c)
          else if l == "       / LOCATION" then Just (StateLocation id : ss, c)
               else if l == "       / SERVICES"
                    then Just (StateServices id : ss, c)
                    else if l == "       / SINGLETONS"
                         then Just (StateSingletons id : ss, c)
                         else if l == "     / Loader"
                              then Just (ss, c)
                              else Nothing
      parseLine'' ss@(StateActivator id : _) l =
          case l =~~~ "^                Data = \"(.+)\"$" of
            Just [n] -> Just (ss, setActivator c id n)
            _ -> Nothing
      parseLine'' ss@(StateLocation id : _) l =
          case l =~~~ "^                Data = \"(.+)\"$" of
            Just [n] -> Just (ss, setLocation c id n)
            _ -> Nothing
      parseLine'' ss@(StateServices id : _) l =
          case l =~~~ "^         / (.+)$" of
            Just [n] -> Just (ss, addService c id n)
            _ -> Nothing
      parseLine'' ss@(StateSingletons id : _) l =
          case l =~~~ "^         / (.+)$" of
            Just [n] -> Just (ss, addSingleton c id n)
            _ -> Nothing
 
parse :: IO Impls
parse = parse' [StateStart] empty
    where parse' s c = do (s', c') <- parseLine s c
                          if null s' then return c' else parse' s' c'
 
toComps :: Impls -> Comps
toComps = foldWithKey toComp empty
    where toComp id impl cs = alter alt (location impl) cs
              where alt Nothing =
                        Just $ Comp { loader = activator impl,
                                      impls = singleton id impl }
                    alt (Just c) | loader c == activator impl =
                        Just $ c { impls = insert id impl $ impls c }
 
xmlComponent :: (String, Comp) -> IO ()
xmlComponent (uri, comp) = do
  putStrLn $
    "  <component loader=\"" ++ loader comp ++ "\" uri=\"" ++ uri ++ "\">"
  mapM_ xmlImplementation $ toList $ impls comp
  putStrLn "  </component>"
 
xmlImplementation :: (String, Impl) -> IO ()
xmlImplementation (id, impl) = do
  putStrLn $ "    <implementation name=\"" ++ id ++ "\">"
  mapM_ xmlService $ sort $ services impl
  mapM_ xmlSingleton $ sort $ singletons impl
  putStrLn "    </implementation>"
 
xmlService :: String -> IO ()
xmlService n = putStrLn $ "      <service name=\"" ++ n ++ "\"/>"
 
xmlSingleton :: String -> IO ()
xmlSingleton n = putStrLn $ "      <singleton name=\"" ++ n ++ "\"/>"
 
main :: IO ()
main = do impls <- parse
          let comps = toComps impls
          putStrLn "<?xml version=\"1.0\"?>"
          putStrLn "<components xmlns=\"http://openoffice.org/2010/uno-components\">"
          mapM_ xmlComponent $ toList comps
          putStrLn "</components>"

That nothing broke during translation can be verified (within the Hamburg environment) with the below throwaway Bash script:

#!/bin/bash
set -e -o pipefail
 
mkdir out
 
function regview () {
 LD_LIBRARY_PATH="${SOURCE_ROOT_DIR?}"/"${INPATH?}"/lib"$UPDMINOREXT" \
  "${SOURCE_ROOT_DIR?}"/"${INPATH?}"/bin"$UPDMINOREXT"/regview "$@"
}
 
function extract () {
 regview "${1?}"/"${2?}"/misc/services.rdb > \
  out/regview-"${WORK_STAMP?}"_"${UPDMINOR?}"-"$i"-ure-services.rdb
 regview "${1?}"/"${3?}"/program/services.rdb > \
  out/regview-"${WORK_STAMP?}"_"${UPDMINOR?}"-"$i"-basis-services.rdb
 regview "${1?}"/"${3?}"/program/legacy_binfilters.rdb > \
  out/regview-"${WORK_STAMP?}"_"${UPDMINOR?}"-"$i"-basis-legacy_binfilters.rdb
}
 
for i in {unxlngi6,unxsoli4,unxsols4}{,.pro} unxlngx6.pro; do
 extract \
  /so/ws/"${WORK_STAMP?}"/"$i"/installation."${UPDMINOR?}"/opt/openoffice.org \
  ure/share basis3.4
done
 
for i in unxmacxi{,.pro}; do
 extract \
  /so/ws/"${WORK_STAMP?}"/"$i"/installation."${UPDMINOR?}"/opt/OpenOffice.org.app/Contents \
  basis-link/ure-link/share basis-link
done
 
for i in wntmsci12{,.pro}; do
 mkdir tmp
 (cd tmp && unzip -q \
  /so/install/"$i"/OpenOffice/archive/"${WORK_STAMP?}"_"${UPDMINOR?}"_native_packed-*_en-US.*/OOo_3.4.0_Win_x86_install-arc_en-US.zip)
 extract "$PWD"/tmp/OOo_3.4.0_Win_x86_install-arc_en-US/OpenOffice.org\ 3 URE \
  Basis
 rm -r tmp
done
 
for i in out/regview-*; do
 ../haskell/components < "$i" > out/processed"${i#out/regview}"
done
 
function filter () {
 sed -e 's/<component /\n<component /g' \
  -e 's:</components>:\n</components>:' | grep '^<component ' | sort | \
  sed -e 's/<implementation /\n<implementation /g'
}
 
function compare () {
 printf '\n%s %s:\n' "${1?}" "${2?}"
 diff \
  <(cat out/processed-"${WORK_STAMP?}"_"${UPDMINOR?}"-"${1?}"-"${2?}".rdb | \
    sed -e 's/^ *//' | tr -d '\n' | filter) \
  <(cat "${SOURCE_ROOT_DIR?}"/"${1?}"/xml/"${3?}".rdb | filter) || [ $? -eq 1 ]
}
 
for i in {unxlngi6,unxmacxi,unxsoli4,unxsols4,wntmsci12}{,.pro} unxlngx6.pro
do
 compare "$i" ure-services ure/services
 compare "$i" basis-services services
 compare "$i" basis-legacy_binfilters legacy_binfilters
done

Extensions

For backwards compatibility, UNO components included in extensions that shall still work with older OOo versions must continue to use active registration. The components that are probably affected (and which have not been translated to passive registration above) are exactly those in modules mysqlc, reportbuilder, sdext, and swext.

For Extension Manager live deployment to work reliably, extension jar components must not be themselves Java type libraries (i.e., must contain a UNO-Type-Path manifest entry, and that entry must not mention <>).

Removals

• solenv/src/component.map and solenv/src/unloadablecomponent.map no longer mention component_writeInfo (a new solenv/src/reg-component.map is a copy of the original component.map, including component_writeInfo).
• comphelper/servicedecl.hxx no longer implements component_writeInfo functionality.
• regcomplazy is gone.
• STARREGISTRY, UNO_COMPONENT, RegistryID, NativeServicesURLPrefix, JavaServicesURLPrefix, and Regmergefile are gone from scp2 (and thus also UNO_JAR_FILE, UNO_JAR_FILE_PATCH, PACKED_UNO_LIB_FILE_BODY, PACKED_UNO_LIB_FILE_BODY_PATCH, STD_UNO_LIB_FILE, STD_UNO_LIB_FILE_PATCH, SPECIAL_UNO_LIB_FILE, SPECIAL_UNO_LIB_FILE_PATCH, SPECIAL_UNO_COMPONENT_LIB_FILE, SPECIAL_UNO_COMPONENT_LIB_FILE_PATCH, and SPECIAL_UNO_NO_WARNING_IF_NOT_EXISTS_FILE from scp2/inc/macros.inc; replacements are SPECIAL_COMPONENT_LIB_FILE and SPECIAL_COMPONENT_LIB_FILE_PATCH).
• SERVICESPROJEKT is gone from instsetoo_native.

Open Items

• Extend XML format to include information for component_getImplementationEnvironment/component_canUnload, and simplify component_getFactory (for each implementation, record in XML a symbol exported from the dynamic library, representing a function that returns an instance of the given implementation).

• Look for overlap with “IDL-XML-Converter: A Package for Transforming IDL into XML”.

• Today there is a single basis-layer services.rdb that also includes information about UNO components from optional installation packages, which might not be there at runtime. It would be better to have this split into multiple files, one for each (optional) installation unit. The question is how to then list exactly the actually present files in URE_MORE_SERVICES in basis-layer fundamentalbasis ini file.