Build Environment Effort/External source tarballs

Build Environment Effort
Quick Navigation Customers Build Performance Elimination of Non-Pro Dependencies Module Problems Split Build Scalability HID List New Build System Requirements External Source Tarballs ToDo Build System Analysis Zen of gbuild Gbuild Bootstrapping Module Migration Module Migration Tips Build System Analysis:capstone2013 windows build Capstone 2013 Client Requirements Document Capstone 2013 Technology Review & Implementation Document
About this template

Integration of external source tarballs in our build process

General assumptions

Most available tarballs can be built (theoretically) by following three steps:

1. unpack
2. configure
3. build

Taking this as the generic way some targets are implemented in our makefiles to ease the integration of external projects.

What can be done?

With a single dmake you can unpack, patch, configure and build an alien tarball, provided that the according makefile.mk is correct. Each of these steps is skipped when run successfully once. The files in "<module>/<platform>/misc/build" reflect the already done parts and can be deleted to force rerunning a particular step.

Additionally there are some targets for common use actions which can be used with "dmake TARGET":

• create_clean: unpack tarball without modifying it.

• patch: apply the patchfiles from "PATCH_FILES" to source tree. The tree is created if not already existing.

• create_patch: a patchfile (named <tarball basename>-newpatch-rename_me +.patch) is created, reflecting all new changes to files from the tarball, done additionally to previous patches. Note that you still have to give that patch a sensible name and add it to PATCH_FILES Additional files have to be set in the variable "ADDITIONAL_FILES" to get into the patch.

Where are those tarballs?

The tarballs reside outside of the main source code. Setting up a build environment will set an environment variable named TARFILE_LOCATION which is typically set to ext_sources. The tarballs are stored with their checksum as "<md5-checksum>-tarfilename" (see also TARFILE_MD5 in the example).

To initially fill this directory, a script called fetch_tarballs.sh is called. The only parameter is the list of all required tarfiles for this specific revision, found in the repository root. Thus

fetch_tarballs.sh external_deps.lst

checks for the required tarballs and downloads them from the URL given in the first line of this list. This step by default is done in the bootstrap step of setting up an environment.

How to get new tarballs upstream?

To get new archives to the download directory, please contact the lead of the external project, Martin Hollmichel.

Update the file external_deps.lst in the repository root with the file name of the new tarball exactly as it appears in $TARFILE_LOCATION.

What's missing?

• Dependencies are only between the supported steps or tarball internal.

• Known feature like "dmake debug" only work in modules that add dmake makefiles to build the external sources.
  

• switching to a gcc environment on win32 .

• external projects are build as one job. There is no parallelization the build the external project -- this might create a bottleneck for highly parallelized builds, if the external project is big.

How to write a makefile.mk for an external tarball

This example is taken from the so_berkeley project.

#*************************************************************************

PRJ=.

PRJNAME=so_berkeleydb
TARGET=so_berkeleydb

# --- Settings


.INCLUDE : settings.mk


# --- Files

# Basename of the tarball which is supposed to reside in a direcrory named pointed to by the environment variable TARFILE_LOCATION. 
TARFILE_NAME=db-3.2.9

# md5 checksum of the tarball 
TARFILE_MD5=d70951c80dabecc2892c919ff5d07172

# Name of the patch file if needed. Used to apply patches to the original tarball needed 
# to build (e.g. for win32...). 

PATCH_FILES=db-3.2.9.patch

.IF "$(GUI)"=="UNX"

# sometimes it makes sense to call the configure command from a different directory. 
CONFIGURE_DIR=out

#relative to CONFIGURE_DIR

# command to configure package and switches. If no "CONFIGURE_ACTION" is supplied 
# all other configure variables are ignored. 

CONFIGURE_ACTION=..$/dist$/configure

CONFIGURE_FLAGS=--enable-cxx --enable-java --enable-dynamic --enable-shared

# same as with configure. 
BUILD_DIR=$(CONFIGURE_DIR)
BUILD_ACTION=make

# copy libraries to the "lib" directory in the local output tree known from own projects. 
OUT2LIB=$(BUILD_DIR)$/.libs$/libdb*.so

# same as above. copy to $(BIN) 
OUT2BIN=java$/classes$/db.jar

.ENDIF # "$(GUI)"=="UNX"

.IF "$(GUI)"=="WNT"

BUILD_DIR=build_win32 

# build a msdev project without GUI. 
BUILD_ACTION=msdev Berkeley_DB.dsw /useenv /MAKE "db_buildall - RELEASE" /MAKE "db_java - RELEASE"

OUT2BIN=java$/classes$/db.jar \
    $(BUILD_DIR)$/Release$/libdb_java32.dll \
    $(BUILD_DIR)$/Release$/libdb32.dll

OUT2LIB= \
    $(BUILD_DIR)$/Release$/libdb_java32.lib \
    $(BUILD_DIR)$/Release$/libdb32.lib

.ENDIF # "$(GUI)"=="WNT"

# make the created patch contain the new files comments.txt 
ADDITIONAL_FILES= build_w32$/comment.txt

OUT2INC= \
    $(BUILD_DIR)$/db.h \
    include$/db_185.h \
    include$/db_cxx.h

OUT2CLASS=java$/classes$/db.jar

# --- Targets

# contains some nessecary variables. 
.INCLUDE : set_ext.mk
.INCLUDE : target.mk

# contains the "magic" to create targets out of the set variables. 
.INCLUDE : tg_ext.mk