Passive Component Registration

From Apache OpenOffice Wiki
Revision as of 12:30, 8 September 2010 by Sb (Talk | contribs)

Jump to: navigation, search

Motivation

UNO components in the various language bindings (dynamic libraries for binary/C++ UNO, jars for Java UNO, etc.) have mechanisms to actively report information about the services and singletons they offer (exported symbol component_writeInfo for dynamic libraries, public static __writeRegistryServiceInfo method for jars, etc.). This active component registration has various drawbacks:

  • Calls to actively register the components (regcomp) need to be made in carefully crafted environments, ensuring that the relevant UNO components can actually be loaded and their code executed. Especially for Java and Python components, this is nontrivial.
    • At OOo build time, when a services.rdb is assembled in instsetoo_native.
    • At OOo installation time, when bundled extensions that contain UNO components are registered.
    • At OOo run time, when extensions that contain UNO components are registered or revoked.
  • Loading the UNO components to actively execute their code can be time consuming (especially if the components' functionality is not otherwise needed, like when assembling a services.rdb at build time).

Therefore, it is sought to replace active component registration with a passive approach, where a UNO component is accompanied by a representation of the relevant information that can be obtained without running the component's code.

An XML Format

The canonic choice for representing such information these days is XML. The necessary information is encoded as follows:

An XML document encoding information about multiple UNO components (i.e., corresponding to a services.rdb) has a root element with local name components and namespace URI http://openoffice.org/2010/uno-components.

The components root element contains zero or more elements with local name component and namespace URI http://openoffice.org/2010/uno-components, each with an attribute with unprefixed name uri, the value of which is a URI (in OOo-internal form) denoting the relevant UNO component (and which may be a vnd.sun.star.expand URL), and an attribute with unprefixed name loader, the value of which is the name of the relevant UNO service with which to load the component (e.g., com.sun.star.loader.SharedLibrary or com.sun.star.loader.Java2). Each component element in turn contains zero or more service elements, followed by zero or more singleton elements.

Each element with local name service or singleton and namespace URI http://openoffice.org/2010/uno-components has an attribute with unprefixed name name, the value of which is the name of the respective service or singleton.

(The information provided by active component registration contains redundancies that are dropped from this XML format. For one, UNO components are listed in an IMPLEMENTATIONS section and each service and singleton is again listed in a SERVICES resp. SINGLETONS section. For another, each singleton specifies the service implementing it.)

There was already a somewhat similar textual format for regcomplazy in use, which however was based on ini-file syntax (but with irregularities, containing lines that do not match the key=value format) instead of XML. This has been superseded by the approach described here, and regcomplazy has been removed.

As we will see later, within the OOo build system there are also XML files describing single UNO components, and which have a component element instead of a components element as their root element, but otherwise use the same structure as the above XML format.

XML at Run Time

There was a choice whether to either take the XML files and inject them into registry-format services.rdb files (i.e., to still call regcomp resp. unopkg at build resp. installation time), or to use the XML files directly at run time. I decided for the latter.

The configmgr re-write showed that it is acceptable performance-wise to read a handful of XML files during OOo start up, using an XML reader specifically written for that task. So it should be acceptable to replace the existing registry-format services.rdb files (for a start, the URE layer services.rdb and the basis layer services.rdb and legacy_binfilters.rdb) with XML files.

The advantages are easier and potentially faster creation (no need for regcomp like code any more), and easier manipulation (due to the textual format). Also, the registry format has a known design defect, in that it cannot handle arbitrary combinations of key names.

Work in Progress

Work is done on CWS sb129  , currently as a stack of MQ patches:

simpleregistry-cleanup cleans up the stoc/source/simpleregistry code so that subsequent patches can better modify it.

xmlreader extracts the XmlReader code from configmgr to a new xmlreader URE module, so that it can be reused from stoc. An open problem is how to version its C++ ABI (currently done via symbol visibility, which does not allow for versioning).

textualservices extends the implementation of com.sun.star.registry.SimpleRegistry in stoc/source/simpleregistry, so that it can internally also read XML files in addition to registry files, and present them to its clients as if they had the structure of registry-format services.rdb files. This appeared to be the simplest approach for now, given how deep the knowledge about registry-format files is rooted in the code base (see, for example, the functionality in cppuhelper/bootstrap.hxx that bootstraps a UNO environment based on rdb files); it is still subject to change, however.

passive replaces the active registration of all of OOo's UNO components with passive registration. The resulting XML files are still called services.rdb etc. (instead of, say, services.xml), mainly because [ure/source/README] declares the URE services.rdb as part of the published URE interface (acknowledging only its existence, not its internal format, and especially warning against registering anything into it).

.component Files

For each UNO component implementation in the OOo code base that does not implement a UNO component that is included in backwards-compatible extensions (see below), any existing active registration code is removed, and instead a X.component file is added. That file describes component X in the above XML format, with a component root element that lacks the uri attribute (because it is platform dependent, e.g., libvclli.so vs. vclmi.dll). Then, a makefile.mk rule uses new solenv/bin/createcomponent.xslt to turn this into a proper .component file including uri. New postprocess/packcomponents/makefile.mk uses new solenv/bin/packcomponents.xslt to bundle those .component files into the basis-layer services.rdb (similarly, URE-layer services.rdb is assembled in ure/source/makefile.mk and basis-layer legacy_binfilters.rdb is assembled in binfilter/util/makefile.mk).

The corresponding information about UNO component registration was removed from the scp2 data. One problem is that what components were actually registered into a registry-format services.rdb in instsetoo_native depended on what scp2 File definitions are visible when building a product. This varies among products (especially between OpenOffice.org and Oracle Open Office), so that different products used to silently produce different basis-layer services.rdb files (even though this must not be). With the new, more static approach of assembling services.rdb files, this needs to be addressed properly (see TODOs in postprocess/packcomponents/makefile.mk).

The below throwaway Haskell script was used to translate the regview output of an existing registry-format services.rdb into XML format, copying the resulting component elements into the individual .component files:

-- Makefile:
--  .PHONY: components
--  components:
--   ghc --make -o $@ components.hs
--
-- Extract components XML data from regview output:
--
-- $ regview ... | components > ...
 
{-# LANGUAGE ScopedTypeVariables #-}
 
module Main (main) where
 
import Data.Map (Map, alter, empty, foldWithKey, insert, singleton, toList)
import Data.List (sort)
import Text.Regex.Posix ((=~))
 
type Impls = Map String Impl -- keyed on implementation name
data Impl =
    Impl { activator :: String, location :: String, services :: [String],
           singletons :: [String] }
    deriving Show
 
type Comps = Map String Comp -- keyed on URI
data Comp = Comp { loader :: String, impls :: Impls }
    deriving Show
 
data State = StateStart | StateImplementations | StateImplementation String
           | StateActivator String | StateLocation String | StateServices String
           | StateSingletons String
 
(=~~~) :: String -> String -> Maybe [String]
a =~~~ b = if c /= "" then Just ds else Nothing
    where (_::String, c::String, _::String, ds) = a =~ b
 
getImpl :: Maybe Impl -> Impl
getImpl (Just impl) = impl
getImpl Nothing = Impl undefined undefined [] []
 
setActivator :: Impls -> String -> String -> Impls
setActivator c id n = alter alt id c
    where alt x = let i = getImpl x in Just $ i { activator = n }
 
setLocation :: Impls -> String -> String -> Impls
setLocation c id n = alter alt id c
    where alt x = let i = getImpl x in Just $ i { location = n }
 
addService :: Impls -> String -> String -> Impls
addService c id n = alter alt id c
    where alt x = let i = getImpl x in Just $ i { services = n : services i }
 
addSingleton :: Impls -> String -> String -> Impls
addSingleton c id n = alter alt id c
    where alt x = let i = getImpl x
                  in Just $ i { singletons = n : singletons i }
 
parseLine :: [State] -> Impls -> IO ([State], Impls)
parseLine s c = do l <- getLine
                   return $ parseLine' s l
    where
      parseLine' ss@(_ : sr) l =
          case parseLine'' ss l of
            Just a -> a
            _ -> parseLine' sr l
      parseLine' [] l = (s, c)
      parseLine'' ss@(StateStart : _) l =
          if l == " / IMPLEMENTATIONS" then Just (StateImplementations : ss, c)
          else Nothing
      parseLine'' ss@(StateImplementations : _) l =
          case l =~~~ "^   / (.+)$" of
            Just [id] -> Just (StateImplementation id : ss, c)
            _ -> case l =~~~ "^ / " of
                   Just _ -> Just ([], c)
                   _ -> Nothing
      parseLine'' ss@(StateImplementation id : _) l =
          if l == "       / ACTIVATOR" then Just (StateActivator id : ss, c)
          else if l == "       / LOCATION" then Just (StateLocation id : ss, c)
               else if l == "       / SERVICES"
                    then Just (StateServices id : ss, c)
                    else if l == "       / SINGLETONS"
                         then Just (StateSingletons id : ss, c)
                         else if l == "     / Loader"
                              then Just (ss, c)
                              else Nothing
      parseLine'' ss@(StateActivator id : _) l =
          case l =~~~ "^                Data = \"(.+)\"$" of
            Just [n] -> Just (ss, setActivator c id n)
            _ -> Nothing
      parseLine'' ss@(StateLocation id : _) l =
          case l =~~~ "^                Data = \"(.+)\"$" of
            Just [n] -> Just (ss, setLocation c id n)
            _ -> Nothing
      parseLine'' ss@(StateServices id : _) l =
          case l =~~~ "^         / (.+)$" of
            Just [n] -> Just (ss, addService c id n)
            _ -> Nothing
      parseLine'' ss@(StateSingletons id : _) l =
          case l =~~~ "^         / (.+)$" of
            Just [n] -> Just (ss, addSingleton c id n)
            _ -> Nothing
 
parse :: IO Impls
parse = parse' [StateStart] empty
    where parse' s c = do (s', c') <- parseLine s c
                          if null s' then return c' else parse' s' c'
 
toComps :: Impls -> Comps
toComps = foldWithKey toComp empty
    where toComp id impl cs = alter alt (location impl) cs
              where alt Nothing =
                        Just $ Comp { loader = activator impl,
                                      impls = singleton id impl }
                    alt (Just c) | loader c == activator impl =
                        Just $ c { impls = insert id impl $ impls c }
 
xmlComponent :: (String, Comp) -> IO ()
xmlComponent (uri, comp) = do
  putStrLn $
    "  <component loader=\"" ++ loader comp ++ "\" uri=\"" ++ uri ++ "\">"
  mapM_ xmlImplementation $ toList $ impls comp
  putStrLn "  </component>"
 
xmlImplementation :: (String, Impl) -> IO ()
xmlImplementation (id, impl) = do
  putStrLn $ "    <implementation name=\"" ++ id ++ "\">"
  mapM_ xmlService $ sort $ services impl
  mapM_ xmlSingleton $ sort $ singletons impl
  putStrLn "    </implementation>"
 
xmlService :: String -> IO ()
xmlService n = putStrLn $ "      <service name=\"" ++ n ++ "\"/>"
 
xmlSingleton :: String -> IO ()
xmlSingleton n = putStrLn $ "      <singleton name=\"" ++ n ++ "\"/>"
 
main :: IO ()
main = do impls <- parse
          let comps = toComps impls
          putStrLn "<?xml version=\"1.0\"?>"
          putStrLn "<components xmlns=\"http://openoffice.org/2010/uno-components\">"
          mapM_ xmlComponent $ toList comps
          putStrLn "</components>"

That nothing broke during translation can be verified (within the Hamburg environment) with the below throwaway Bash script:

#!/bin/bash
set -e -o pipefail
 
mkdir out
 
for i in {unxlngi6,unxsoli4,unxsols4}{,.pro} unxlngx6.pro; do
  LD_LIBRARY_PATH=/so/ws/DEV300/unxlngi6.pro/lib.m87 /so/ws/DEV300/unxlngi6.pro/bin.m87/regview /so/ws/DEV300/$i/installation.m87/opt/openoffice.org/ure/share/misc/services.rdb > out/regview-DEV300_m87-$i-ure-services.rdb
  LD_LIBRARY_PATH=/so/ws/DEV300/unxlngi6.pro/lib.m87 /so/ws/DEV300/unxlngi6.pro/bin.m87/regview /so/ws/DEV300/$i/installation.m87/opt/openoffice.org/basis3.4/program/services.rdb > out/regview-DEV300_m87-$i-basis-services.rdb
  LD_LIBRARY_PATH=/so/ws/DEV300/unxlngi6.pro/lib.m87 /so/ws/DEV300/unxlngi6.pro/bin.m87/regview /so/ws/DEV300/$i/installation.m87/opt/openoffice.org/basis3.4/program/legacy_binfilters.rdb > out/regview-DEV300_m87-$i-basis-legacy_binfilters.rdb
done
 
for i in unxmacxi{,.pro}; do
  LD_LIBRARY_PATH=/so/ws/DEV300/unxlngi6.pro/lib.m87 /so/ws/DEV300/unxlngi6.pro/bin.m87/regview /so/ws/DEV300/$i/installation.m87/opt/OpenOffice.org.app/Contents/basis-link/ure-link/share/misc/services.rdb > out/regview-DEV300_m87-$i-ure-services.rdb
  LD_LIBRARY_PATH=/so/ws/DEV300/unxlngi6.pro/lib.m87 /so/ws/DEV300/unxlngi6.pro/bin.m87/regview /so/ws/DEV300/$i/installation.m87/opt/OpenOffice.org.app/Contents/basis-link/program/services.rdb > out/regview-DEV300_m87-$i-basis-services.rdb
  LD_LIBRARY_PATH=/so/ws/DEV300/unxlngi6.pro/lib.m87 /so/ws/DEV300/unxlngi6.pro/bin.m87/regview /so/ws/DEV300/$i/installation.m87/opt/OpenOffice.org.app/Contents/basis-link/program/legacy_binfilters.rdb > out/regview-DEV300_m87-$i-basis-legacy_binfilters.rdb
done
 
for i in wntmsci12{,.pro}; do
  mkdir tmp
  (cd tmp && unzip /so/install/$i/OpenOffice/archive/DEV300_m87_native_packed-1_en-US.9522/OOo_3.4.0_Win_x86_install-arc_en-US.zip)
  LD_LIBRARY_PATH=/so/ws/DEV300/unxlngi6.pro/lib.m87 /so/ws/DEV300/unxlngi6.pro/bin.m87/regview $PWD/tmp/OOo_3.4.0_Win_x86_install-arc_en-US/OpenOffice.org\ 3/URE/misc/services.rdb > out/regview-DEV300_m87-$i-ure-services.rdb
  LD_LIBRARY_PATH=/so/ws/DEV300/unxlngi6.pro/lib.m87 /so/ws/DEV300/unxlngi6.pro/bin.m87/regview $PWD/tmp/OOo_3.4.0_Win_x86_install-arc_en-US/OpenOffice.org\ 3/Basis/program/services.rdb > out/regview-DEV300_m87-$i-basis-services.rdb
  LD_LIBRARY_PATH=/so/ws/DEV300/unxlngi6.pro/lib.m87 /so/ws/DEV300/unxlngi6.pro/bin.m87/regview $PWD/tmp/OOo_3.4.0_Win_x86_install-arc_en-US/OpenOffice.org\ 3/Basis/program/legacy_binfilters.rdb > out/regview-DEV300_m87-$i-basis-legacy_binfilters.rdb
  rm -r tmp
done
 
for i in out/regview-*; do
  ../haskell/components < "$i" > out/processed"${i#out/regview}"
done
 
for i in {unxlngi6,unxmacxi,unxsoli4,unxsols4,wntmsci12}{,.pro} unxlngx6; do
  printf '\n%s ure-services:\n' $i
  diff <(cat /cws/so-cwsserv02/sb129/DEV300/$i/xml/ure/services.rdb | sed -e 's/<component /\n<component /g' -e 's:</components>:\n</components>:' | grep '^<component ' | sort) <(cat out/processed-DEV300_m87-$i-ure-services.rdb | sed -e 's/^ *//' | tr -d '\n' | sed -e 's/<component /\n<component /g' -e 's:</components>:\n</components>:' | grep '^<component ' | sort) || [ $? -eq 1 ]
  printf '\n%s basis-services:\n' $i
  diff <(cat /cws/so-cwsserv02/sb129/DEV300/$i/xml/services.rdb | sed -e 's/<component /\n<component /g' -e 's:</components>:\n</components>:' | grep '^<component ' | sort) <(cat out/processed-DEV300_m87-$i-basis-services.rdb | sed -e 's/^ *//' | tr -d '\n' | sed -e 's/<component /\n<component /g' -e 's:</components>:\n</components>:' | grep '^<component ' | sort) || [ $? -eq 1 ]
  printf '\n%s legacy_binfilters-services:\n' $i
  diff <(cat /cws/so-cwsserv02/sb129/DEV300/$i/xml/legacy_binfilters.rdb | sed -e 's/<component /\n<component /g' -e 's:</components>:\n</components>:' | grep '^<component ' | sort) <(cat out/processed-DEV300_m87-$i-basis-legacy_binfilters.rdb | sed -e 's/^ *//' | tr -d '\n' | sed -e 's/<component /\n<component /g' -e 's:</components>:\n</components>:' | grep '^<component ' | sort) || [ $? -eq 1 ]
done

Extensions

For backwards compatibility, UNO components included in extensions that shall still work with older OOo versions must continue to use active registration. The components that are probably affected (and which have not been translated to passive registration above) are exactly those in modules mysqlc, reportbuilder, sdext, and swext.

How passive registration of UNO components in extensions will work is still open.

Removals

  • solenv/src/component.map and solenv/src/unloadablecomponent.map no longer mention component_writeInfo (a new solenv/src/reg-component.map is a copy of the original component.map, including component_writeInfo).
  • comphelper/servicedecl.hxx no longer implements component_writeInfo functionality.
  • regcomplazy is gone.
  • STARREGISTRY, UNO_COMPONENT, RegistryID, NativeServicesURLPrefix, JavaServicesURLPrefix, and Regmergefile are gone from scp2 (and thus also UNO_JAR_FILE, UNO_JAR_FILE_PATCH, PACKED_UNO_LIB_FILE_BODY, PACKED_UNO_LIB_FILE_BODY_PATCH, STD_UNO_LIB_FILE, STD_UNO_LIB_FILE_PATCH, SPECIAL_UNO_LIB_FILE, SPECIAL_UNO_LIB_FILE_PATCH, SPECIAL_UNO_COMPONENT_LIB_FILE, SPECIAL_UNO_COMPONENT_LIB_FILE_PATCH, and SPECIAL_UNO_NO_WARNING_IF_NOT_EXISTS_FILE from scp2/inc/macros.inc; replacements are SPECIAL_COMPONENT_LIB_FILE and SPECIAL_COMPONENT_LIB_FILE_PATCH).
  • SERVICESPROJEKT is gone from instsetoo_native.

Open Items

  • Extend XML format to include information for component_getImplementationEnvironment/component_canUnload, and simplify component_getFactory (for each implementation, record in XML a symbol exported from the dynamic library, representing a function that returns an instance of the given implementation).
  • Python components.
Personal tools