User:TerryE/phpBB3.0.4 Migration/Detailed Implementation Notes

Detailed Implementation Notes

usOOo server scripts

dumpAllDelta.sh

This script is used to unload the current forums

#! /bin/bash
#
# Do a delta dump of the forums
#
unalias -a
outDir='/opt/coolstack/apache2/htdocs/XXXX' # not real directory name
appRoot="/opt/coolstack/apache2"
psql="psql -U ooo_oucv_admin"
pg_dump="pg_dump -i -U ooo_oucv_admin -x"
 
dumpDB(){
   co="$1"
   db='en'
   test "$co" == "zh" && db='zh'
   echo Dumping $co from database $db
   if test "$2" = "sync" ; then
      tables="acl_groups acl_options acl_roles acl_roles_data acl_users attachments \
         banlist bookmarks confirm disallow drafts forums forums_access forums_track \
         forums_watch groups log moderator_cache poll_options poll_votes posts privmsgs \
         privmsgs_folder privmsgs_rules privmsgs_to profile_fields profile_fields_data \
         profile_fields_lang profile_lang reports reports_reasons search_results \
         sessions sessions_keys sitelist topics topics_posted topics_track topics_watch \
         user_group users warnings"
   else
      tables="`$psql -c "\d" $db  | perl -ne \"/\w+_${co}_(\w+)\s+\| table/ && print \\\$1.' ';\"`"
   fi
   # pg_dump V8.1 doesn't support pattern wildcards on the -t option :-(
   ( for t in $tables; do $pg_dump -t  phpbb_${co}_$t $db ; done ; ) \
      | gzip -c > $outDir/$co.sql.gz
}
 
dumpFiles() {
   co=$1
   timestamp="-newer $outDir/lastCopy.Timestamp"
   avatars="$appRoot/htdocs/$co/forum/images/avatars/upload"
   files="$appRoot/htdocs/$co/forum/files"
   ( cd $avatars ; find . $timestamp -type f ) | sed -e 's!^\./!!' > $outDir/fileList
   ( cd $avatars ; tar cf - -I $outDir/fileList ) > $outDir/avatars_${co}.tar
   ( cd $files ; find . $timestamp -type f ) | sed -e 's!^\./!!' > $outDir/fileList
   ( cd $files ; tar cf - -I $outDir/fileList ) | bzip2 -c > $outDir/files_${co}.tar.bz2
   rm $outDir/fileList
}
 
for co in en es fr hu ja vi zh; do
   echo "Processing $co ..."
   dumpDB $co sync
   dumpFiles $co
done

pullAllDelta.sh

This script is run on the new system to pull the databases:

#! /bin/bash
#
# Pull the delta dump of the forums from u.s.oo.o
#
unalias -a
 
usooo=192.18.196.107
migrationDir="http://$usooo/XXXX" # not real directory name
alias wget=/usr/sfw/bin/wget
 
for co in en es fr hu ja vi zh; do
   wget $migrationDir/avatars_$co.tar
   wget $migrationDir/files_$co.tar.bz2
   wget $migrationDir/$co.sql.gz
done

Patch for the phpBB database_update.php script

The standard phpBB script install/database_update.php is used both to path the DDL to reflect any changes in going from version 3.x to current (in our case 3.0.1 to 3.0.4) and to patch the data content. Because I am using having to use a 3.0.4 MySQL schema as a starting point, I need to comment out the DDL patches but still execute the rest of the script (also since my merge strategy leaves the config tables untouched on the Live synchronisation re-import, I need to force the DB schema version to 3.0.1. Anyway, here is the patch

--- /var/www/phpBB_ref/install/database_update.php      Fri Dec 12 16:20:38 2008
+++ /var/www/phpBB-common/install/database_update.php   Tue May  5 14:46:34 2009
@@@ -679,7 +679,7 @@
 {
        $config['version'] = $debug_from_version;
 }*/
-
+$config['version]='3.0.1';                                                 ### UPGRADE PATCH ###
 echo $lang['PREVIOUS_VERSION'] . ' ::<strong>' . $config['version'] . '</strong><br />';
 echo $lang['UPDATED_VERSION'] . ' :: <strong>' . $updates_to_version . '</strong></p>';

@@ -1166,7 +1166,7 @@
                exit_handler();
        }
 }
-
+if (false) {                                                               ### UPGRADE PATCH ###
 // Schema updates
 ?>
        


@@ -1298,7 +1298,7 @@
 }

 _write_result($no_updates, $errored, $error_ary);
-
+}                                                                          ### UPGRADE PATCH ###
 // Data updates
 $error_ary = array();
 $errored = $no_updates = false;

Standard NL configuration

All instances have the same content and essentially symlink everything but the avatars-load, cache and files directories. This means that all image sets and code changes are common to all versions. This includes the specific changes to the French forum that Bidouille requires (and in fact these are enabled by the existence of a specific match parameter that they use.) This all works because all of the forum configuration (such as the selection of the forum's main logo) is maintained in the forum database, and this database is private to each NL forum. In the same way, the individual styles are cached in the database so the Vietnamese forum can tweak its CSS to remove the underlines from links in the database (this is needed because accents in Vietnamese also lie under the letters and an underline can obscure these changing the meaning of the text).

Hence each forum instance has exactly the same structure, excepting the three content directories:

forum:
  adm
  avatars-upload
  cache
  common.php -> ../../phpBB-common/common.php
  config.php -> ../../phpBB-common/config.php
  cron.php -> ../../phpBB-common/cron.php
  docs -> ../../phpBB-common/docs
  download
  faq.php -> ../../phpBB-common/faq.php
  files
  images -> ../../phpBB-common/images
  includes -> ../../phpBB-common/includes
  index.php -> ../../phpBB-common/index.php
  install -> ../../phpBB-common/install   (*) only set up for database conversion.
  language -> ../../phpBB-common/language
  mcp.php -> ../../phpBB-common/mcp.php
  memberlist.php -> ../../phpBB-common/memberlist.php
  posting.php -> ../../phpBB-common/posting.php
  report.php -> ../../phpBB-common/report.php
  search.php -> ../../phpBB-common/search.php
  store
  style.php -> ../../phpBB-common/style.php
  styles -> ../../phpBB-common/styles
  ucp.php -> ../../phpBB-common/ucp.php
  viewforum.php -> ../../phpBB-common/viewforum.php
  viewonline.php -> ../../phpBB-common/viewonline.php
  viewtopic.php -> ../../phpBB-common/viewtopic.php

forum/adm:
  images -> ../../../phpBB-common/adm/images
  index.php -> ../../../phpBB-common/adm/index.php
  style -> ../../../phpBB-common/adm/style
  swatch.php -> ../../../phpBB-common/adm/swatch.php

forum/avatars-upload:
  <instance specific uploaded avatars go here>

forum/cache:
  index.htm -> ../../../phpBB-common/cache/index.htm
    <instance specific generate cache files go here>

forum/download:
  file.php -> ../../../phpBB-common/download/file.php
  index.htm -> ../../../phpBB-common/download/index.htm

forum/files:
  index.htm -> ../../../phpBB-common/files/index.htm
  <instance specific uploaded attachment files go here>

Even through the databases are private to each forum, I would like to standardise these configurations where possible (for example the list of languages, BBcode extensions, etc.).