User:TerryE/phpBB3.0.4 Migration/Standardisation Strategy

OOo Community Forums Migration Strategy

The current OOo Forums started with the creation of the English forum in Nov 2007 running on phpBB 3.0.0 over a Sun Coolstack platform and PostgreSQL 8.2 running on Solaris x86 infrastructure in the Sun Munich Facility. We subsequently upgraded to phpBB 3.0.1 and added the Hungarian, French, Spanish and Japanese forums. We are now in the process of adding the Vietnamese and Chinese forums. Given the proliferation of forums, the general growth in transactional volumes and the fact that phpBB is now at version 3.0.4, we have decided to upgrade the H/W infrastructure and install a clean rebaseline for all forums, This page describes the process and approach for migration the OOo forums from their current platform to the new installation.

Assumptions and Constraints

• The current forum is running on a number of separate phpBB instances all at version 3.0.1 (but with disjoint code hierachies) over a common PostgreSQL 8.2 on a Solaris/Coolstack platform.
• The content of the forums is largely maintained in the central database, though attachment and image data are maintained in the file separate file hierarchies. All such content will be preserved on migration.
• A new H/W platform will be running Solaris 10, plus Coolstack 1.3.1 including the optional CSKPython package. All instances will use the 32bit CSK packaged MySQL 5.0.
• The target forum configuration will be a largely “out of the box” configuration of phpBB version 3.0.4. We will retain a small number of changes as detailed in section Forum Customisation below.
• The forum will have the phpBB Prosilver template, plus supported languages. All languages will be made avaialble to all instances. The Subsilver will not be supported.
• Supported Languages will be English (UK and US), French, Spanish (Spain, Argentia, Catalan, Valencian), Hungarian, Mandarin (Traditional and Simplified Scripts), Japanese and Vietnamese.
• A common code base will be shared by all instances by symlinking the individual instance hierarchies to a common directory set.
• This will be based on a fresh install of phpBB from the phpBB Downloads reference site.
• Each Language Forum will contain an administration forum, access to which will be limited to trusted moderators and administrators.
• We want to move to as "Zero Admin" an approach as possible. Therefore all routine sysadmin functions will be scripted by means of automated (usually crontab based) scripts.
• Python will be used as the standard scripting Language.

Code Base

• Core codebase from phpBB 3.0.4 Download
• The following Language Packs from http://www.phpbb.com/files/language_packs_30x/

Language	Language files	Prosilver GIFs
English (U.K.)	(built-in)	(built-in)
English (U.S.)	lang_en_us.tar.gz	prosilver_en_us.tar.gz
French	lang_fr.tar.gz	prosilver_fr.tar.gz
Catalan	lang_ca.tar.gz	prosilver_ca.tar.gz
Catalan (Valencian)	lang_ca_pv.tar.gz	prosilver_ca_pv.tar.gz
Spanish	lang_es.tar.gz	prosilver_es.tar.gz
Spanish (Argentina)	lang_es_ar.tar.gz	prosilver_es_ar.tar.gz
Hungarian	lang_hu.tar.gz	prosilver_hu.tar.gz
Mandarin Chinese (Traditional script)	lang_zh_cmn_hant.tar.gz	prosilver_zh_cmn_hant.tar.gz
Mandarin Chinese (Simplified script)	lang_zh_cmn_hans.tar.gz	prosilver_zh_cmn_hans.tar.gz
Japanese	lang_ja.tar.gz	prosilver_ja.tar.gz
Vietnamese	(unsupported translation)	(unsupported translation)

• Note that the Vietnamese is not yet available through the phpBB site.

Overview of the Migration Philosophy

I have written some scripts to do a smart delta of the seven current language-specific directory hierarchies. These have helped me to analyse the current differences. Basically the configurations are an exercise of entropy in action: every config is slightly different, both in terms of language packs and php file modifications. This means than any upgrade of the baseline phpBB package has seven completely different versions. The consequences for long term maintenance are just a killer and this is going to get worse as we add support for extra languages, so we must move to a single common code base for all forums.

Previous to this migration I did a migration of the Virtualbox Community Forums from phpBB V2 to V3.0.4, and I used our OOo configuration and customisations as a starting point. I reviewed all of these customisations in terms of their value and volunteer feedback, and junked any that didn't have a proven benefit. This "junked" list includes my experimental kB patch; heavy OOo customisation of the standard phpBB FAQ customisation; the beginners template; the email link to the forumadmin on all pages. I then aligned all remaining changes into the standard phpBB V3.0.4 to derive a patch of the required changes. This work largely applies to the OOo forums as well (except for the OOo specific branding in the headers and footers).

The main difference between the OOo and VirtualBox configurations is that we maintain a number of separate language forums within OOo. The base software configuration will be shared by all forums and the configuration set ups will be largely consistent across all forums. This will enable a common set of language templates to be maintained across all forums, and hence all languages will be available to the individual language forum admins (though these admins might only chose to enable a subset of languages within their own forum). Certain configuration items are forum specific rather than language specific. An example of this is that an advisory message is displayed in the post form asking users to read the Survival Guide before posting. We will have a Survival Guide in every forum, so this instruction can be in the UCP selected language and the language-specific wording will be the same for every forum, so this text will be in the language template. However, the link to actual guide is forum specific and this will be maintained in the system configuration table.

• Language specifics will be maintained in the language template
• Forum specifics will be maintained in the forum phpBB_XX_config table.

The way that phpBB establishes its root with Apache by examining the current directory of an script that is directly executed through a URI. These are as follows:

• forum: cron.php, faq.php, file.php, index.php, listforums.php, mcp.php, memberlist.php, posting.php, report.php, search.php, style.php, ucp.php, viewforum.php, viewonline.php, viewtopic.php
• forum/adm:index.php, swatch.php
• forum/download: file.php

Each forum will therefore maintain private copies of these directories along with the sub-directories which contain forum-specific files: cache, download, files, images, store as well as the config.php file which defines the database context. The remain subdirectories will be symlinked to a common repository:

• docs -> ../../common/forum/docs
• language -> ../../common/forum/language
• styles -> ../../common/forum/styles
• includes -> ../../common/forum/includes

The individual files in the top level directory, etc., will be symlinked to the common copy:

• forum/common.php -> ../../common/forum/common.php
• forum/cron.php -> ../../common/forum/cron.php
• forum/faq.php -> ../../common/forum/faq.php
• forum/file.php -> ../../common/forum/file.phpx`
• forum/index.php -> ../../common/forum/index.php
• forum/listforums.php -> ../../common/forum/listforums.php
• forum/mcp.php -> ../../common/forum/mcp.php
• forum/memberlist.php -> ../../common/forum/memberlist.php
• forum/posting.php -> ../../common/forum/posting.php
• forum/report.php -> ../../common/forum/report.php
• forum/search.php -> ../../common/forum/search.php
• forum/style.php -> ../../common/forum/style.php
• forum/ucp.php -> ../../common/forum/ucp.php

This may sound inefficient, but thanks to the phpBB caching system (which is local to each forum) the material overhead of this redirection is removed. I need to evaluate the image directory tree and update this section in the light of this evaluation.

Hence the software configuration of the new forum versions will be a clean patched install. Only the forum content will be migrated. This content is largely contained in a set of forum-specific tables, though it also comprises a set of files in the cache, download, files, images andstore directories. I have written a small shell script can dump either all tables relating to a specific language forum or the subset of tables that contain the user and post data. I also have a second python script which converts this PostgreSQL V3.0.1 schema and dump into a MySQL V3.0.4 compatible load

The migration will take place in two passes.

• The first involves building a complete forum set in a virtual LAMP appliance. It's just easier to do this on a LAN connected VM than the bare Solaris H/W in Germany over the internet. Then of the each of the current language databases will be snapped into the forum backups directory, and then wget pulled to the LAMP appliance for loading and configuring into the test system. When I have tested this configuration, I will then transfer it to the OOoTest system in the Sun labs in Germany. At this point the individual language forum leaders will be given sysadmin access to each forum to validate functionality and configuration
• Once these admins have confirmed the system configuration of their respective forums we are then ready to go live. The cut over will involve a process that I developed and tested in the VirtualBox migration which is bring the live forums offline, then overlay the current user and post data onto the target OOoTest server. This involves truncating and replacing the content of the following tables from live. Once this import is complete the forums are available for go-live.

banlist, bots, disallow, log, poll_options, poll_votes, posts, privmsgs, privmsgs_folder, privmsgs_to, topics, topics_posted, topics_watch, user_group, users

Forum Customisation

The delivered customised instances contains the following changes:

• Reorder window title to make multi-tab title more usable. All browser tabs are labelled with their title. The phpBB3 default is to begin the title with the forum name then page type, hence if you have a lot of tabs open they are called openOffi... By putting the specific title first, it makes it a lot easier to pick the correct tab.
• Add Expand option to Code Block. phpBB3 now puts each code block in a scrollable window. This makes it a pain if you want to see all of the code in the browser. This mod adds an Expand option to the code block to allow you to blow it up to full size.
• Tweak Access key codes to remove conflicts. For those that use keyboard accelerators within the broswer, the lack of a shortcut for Preview Post is a pain. This makes <Accel>+P a shortcut for Preview and <Accel>+G a shortcut for graphics.
• Add Active Post count to View Active Topics. This just adds a count of new posts in the Active topics heading.
• Add truncated title of last topic to Forum View. Lots of OOo volunteers asked for this one. It enables you to recognise at a glance if a forum has any new posts in it.
• Customise Headers and Footers to OpenOffice.org branding. Add OOo Community logo and adjust framing colours to match. Add OOo favicon.
• Rebasline to phpBB 3.0.4. The install is a patch file which updates the standard 3.0.4 unpacked tarball.
• Add OpenOffice.org Qs to BB FAQ. The FAQ button goes to a phpBB3 standard FAQ which is all about using phpBB3. I've added a Q0 which points the user to the wiki and forum FAQs for OpenOffice info.
• Add “Request to read Survival Guide before posting” to post form. The posting form now includes an instruction to read the FSG before you post. (This is only shown if you have less than 15 posts).
• Add custom fields do define users, main OS, installed OOo version and supplementary OSs. (This is actually a configuration of standard phpBB functionality.)