BuildSpeedup

BuildSpeedup summary

Make the OO.o build system the coolest on the planet! OpenOffice is one of the largest and most interesting open source projects that exists. The target of the BuildSpeedup project is to make the build system the coolest one around. A reliable and fast build system has several direct and inderect impacts, maybe most importantly it is the first thing a new contributor comes across when they pick up the project. Main main target audience is an contributor running a high-mid level desktop. Secondary target is people running build farms (notably Sun). Specification for Jam build prototype SpecJamBuild.

How to report build profile logs

Thank you for sending in a build profile log. The more logs we have, the easier it is to focus our effort correctly. Here are the steps to producing a build log:

1. Apply the latest patch from http://www.openoffice.org/issues/show_bug.cgi?id=60948 to your source tree
2. in the dmake directory type "make clean; make". In the root directory type "./bootstrap"
3. When building redirect stderr and stdout to the logfile. In tcsh "build --all |& tee myconfigdesc_build.log" will do the trick. You can alternatively use cat instead of tee to speed up the build slightly. You need to build in a single go and in a single process (no parallell building)
4. Check that the log contains markers like "s target 1234 foo.obj". If not, then report
5. Compress the log with zip/gz
6. When you send in your log URL to mailto:kaib@openoffice.org please provide a short description of the build type:
   • Important configure options (like --with-lang=?)
   • OS/compiler/generic hardware description
   • No-change, partial or full build (we are especially looking for full builds)
   • External accelerators like ccache

Thank you again for contributing! --KaiB 16:25, 25 January 2006 (CET)

Profile results

Here are some initial profile results. These were done on a HP development desktop with 4GB of RAM. Due to various reasons there is a margin of error in all results, but the largest culprits are pretty clear.

Full rebuild from scratch

Time spent	Target type (file extension)
33.2%	C/C++ compilation (obj)
27%	dmake/build.pl (time not attributed to any target)
22.0%	Dependency generation (dpcc)
3.7%	SDK installer (sdkoo_en-US)
1.3%	Unknown, build related (last_target)
1.2%	Linking (dll)
1.1%	Openoffice installer (openoffice_en-US)

Running time: 62296 s targets: 45927 s 73%

As a benchmark, a Mozilla Firefox build corrected for differences in source code size takes about 25% of the OO.o build time.

Nothing changed rebuild

Basically profiling the build in a non-changed source tree.

Time spent	Target type (file extension)
54.8%	SDK installer (sdkoo_en-US)
19%	dmake/build.pl (time not attributed to any target)
16.2%	OpenOffice installer (openoffice_en-US)

Running time: 4209 s targets: 3431 s 81%

Notes

This is a workarea for ideas and notes regarding the speeding up of the build process. --KaiB 12:50, 16 January 2006 (CET)

Notes:

• Newest dmake version: cws_src680_dmake43p01)
• solenv/inc define global targets
• potential modules: KaiB: transex3, tools, svx, binfilter - small to huge :)
• helpcontent2 --with-lang=ALL (there are open issues on this already)
• batched compiles
• Interix instead of cygwin
• dependency generation

Existing possibly useful Build Speed related tweaks:

• export nodep=true
• export NO_HIDS=true
• build.pl -P parallel flag
• dmake -P parallel flag and/or MAXPROCESS (cws_src680_dmake43p01 has fixes for some problems there)
• dmake -s flag for less verbose output
• for gcj users targetedaot workspace to ahead of time compile (only) HelpLinker and FCFGMerge java build tools

transex3

Started by tackling this small module. Initial full compile time was 137 sec.

build_profile_analyzer.py

(NB. Move this to an attachment once the Wiki accepts .py files)

# Copyright 2006 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# OO.o build profile analyzer. To get the required patches look at:
# http://wiki.services.openoffice.org/wiki/BuildSpeedup
# Author: Kai Backman, kaib@google.com

import os
import re
import sys

def SortProfileTimes(lhs, rhs):
  if lhs[1] < rhs[1] : return 1
  elif rhs[1] < lhs[1] : return -1
  else : return 0

def ProcessLogFile(file_name, target_durations):
#  print "Processing file: " + file_name
  file = open(file_name)
  start_time = 0
  end_time = 0

  target_start = 0
  target_name = ""
  line = file.readline()
  while line != "" :
    tokens = line[:-1].split(" ")
    line = file.readline()

    if len(tokens) < 2 : continue
    if tokens[1] == "build" :
      if tokens[0] == "s" :
        start_time = int(tokens[2]) * 1000
      elif tokens[0] == "e" :
        end_time = int(tokens[2]) * 1000
    elif tokens[1] == "target" :
      if tokens[0] == "s":
        target_start = int(tokens[2])
        target_name = tokens[3]
      elif tokens[0] == "e" and target_name == tokens[3]:
        target_duration = int(tokens[2]) - target_start
        target_tokens = tokens[3].split(".")
        target_class = target_tokens[len(target_tokens) - 1]
        if target_durations.has_key(target_class) :
          target_durations[target_class] += target_duration
        else :
          target_durations[target_class] = target_duration
  file.close()
  return end_time - start_time
  return 


if __name__ == "__main__":
  target_durations = {}
  total_time = 0
  for file_name in sys.argv[1:] :
    total_time += ProcessLogFile(file_name, target_durations)

  profile = target_durations.items()
  profile.sort(SortProfileTimes)
  total_target_time = 0
  for target_iter in profile :
    target_promille = target_iter[1] * 1000 / total_time
    if target_promille > 0 :
      print "%6.ds %3.d.%1.d%% %s" % (target_iter[1] / 1000, target_promille/10,
                                      target_promille % 10, target_iter[0])
    total_target_time += target_iter[1]
  print "Running time: %d s targets: %d s %d%%" % (total_time / 1000, total_target_time / 1000,
                                                   total_target_time * 100 / total_time)