Difference between revisions of "User:TerryE/Traffic Server Configuration"

Latest revision as of 10:15, 23 August 2011

Why Traffic Server?

Apache Traffic Server is a lightweight, yet high-performance, web proxy cache that improves network efficiency and performance^[1] . Like Squid and Varnish, Traffic Server can be configured as a reverse proxy^[2]. In this mode, it acts as a full surrogate for the back-end wiki with port 80 on the advertised hostname for the wiki resolving to Traffic Server. In doing so this enables the processing of web requests to be offloaded from the PHP and database intensive MediaWiki application.

Traffic Server can be configure to store high frequency cached content in memory, and where content is flush to disk, access will still invovle significantly less physical I/O than the MediaWiki application. Hence permitting a significantly higher throughput for a give CPU and I/O resource constraint. MediaWiki id been designed to integrate closely with such web cache packages and will Traffic Server when a page should be purged from the cache in order to be regenerated. From MediaWiki's point of view, a correctly-configured Traffic Server installation is interchangeable with Squid or Varnish.

The architecture

An example setup of Traffic Server, Apache and MediaWiki on a single server is outlined below. A more complex caching strategy may use multiple web servers behind the same Traffic Server caches (all of which can be made to appear to be a single host) or use independent servers to deliver wiki or image content.

To the outside world, Traffic Server appears to act as the web server. In reality it passes on requests to the Apache web server, but only when necessary. An Apache running on the same server only listens to requests from localhost (127.0.0.1) while Traffic Server only listens to requests on the server's external IP address. Both services run on port 80 without conflict as each is bound to different IP addresses.

Traffic Server 3.0.1

Installation and preparation

The ATS README and INSTALL define the Ubuntu package dependencies, so before doing the build the following packages were installed:

sudo apt-get install  autoconf  automake libtool g++ libssl-dev tcl-dev expat libexpat-dev libpcre3-dev libcap-dev

and the system group / user was added for an ats-data account much as Apache server on Debian is configured to use www-data. The package was then installed by executing the following from the kit directory. The configure options mean that the kit is installed with a standard Debian layout and using the ats-service account:

./configure --enable-layout=Debian --with-user=ats-data --with-group=ats-data
make
sudo make install

ATS uses both a memory and a disk cache to buffer content hierarchically. Clearly, memory is used for high frequency content but it is still important that any physical I/O overheads for accession disk-based content are kept to an absolute minimum. Like many database packages, ATS recommends the use of a RAW partition for this disk cache and this needs to be configured. As we use LVM2 on the VM, it is easy to set up the partition

lvcreate -L 2G -C y -n ooo-wiki-TScache  ooo-wiki-data-lvgroup      # Create the ATS cache LV
modprobe raw                                                        # Load the raw device driver
sudo bash -c "echo raw >> /etc/modules"                             # And make sure that it is loaded on reboot
sudo raw /dev/raw/raw1 /dev/ooo-wiki-data-lvgroup/ooo-wiki-TScache  # Map the raw1 device to the LV
sudo chmod 660 /dev/raw/raw1                                        #   then change RW access 
sudo chown ats-data:ats-data /dev/raw/raw1                          #   to ats-data

However, automatically adding this udev enumeration of an LVM mapped raw device ra is complex and can only be done by changing existing /lib/udev generators, so I recommend a KISS approach for now. These have to be executed prior to starting ATS, and as ATS uses a traditional System V init script rather than upstart, I have also adopted this convention and created a tiny trafficserver-raw script to do this. I leave this to an exercise for the reader. One this has been done, we need to hook ATS into the rc startup system:

sudo cp $WHATEVERYOURWOKINGDIRIS/trafficserver-raw /etc/init.d     # Hook the ATS raw helper start script into init.d
sudo update-rc.d trafficserver-raw start 91 2 3 4 5 .              # but is only started 
sudo ln -s /usr/bin/trafficserver /etc/init.d/trafficserver        # Hook the trafficserver start/stop script 
sudo update-rc.d trafficserver start 93 2 3 4 5 . stop 07 0 1 6 .  # ATS is started after Apache and the raw helper
 
sudo mkdir /x1/var_log_trafficserver                               # You need to set up the log directory.  ATS doesn't do this
sudo chown ats-data:ats-data /x1/var_log_trafficserver             # And it needs to be owned by ats-data or the logging will fail
sudo ln -s /x1/var_log_trafficserver /var/log/trafficserver        # Link it into the Debian standard Logging directory structure

Configuration

The ATS Administrator's Guide discusses two simple methods of defining the configuration.^[3]. However, the package also provides Perl modules to facilitate configuration for those admins familiar with using Perl, so I have used this. The baseline configuration is as folllows:

#
# In this configuration, Apache Traffic Servicer (ATS) is configured as an HTTP reverse proxy 
# connected via the IP looback connector (127.0.0.1) to an in-VM Apache service which is running a
# dedicated MediaWiki application to serve the OOo wiki. The VM is single-core with the application
# typically services ~5-30 page requests per minute, plus associated image/CSS/JS requests needed
# to render each page. For this type of configuration/load, the default ATS requires few changes.   
#
# As ATS provides a Perl Module to facilitate such changes, I have used this, and only four config
# files need additions / changes.
#
BEGIN {push @INC, "/x1/wiki-kits/trafficserver-3.0.1/contrib/perl/ConfigMgmt/lib"; }
 
use strict;
use English;
use Sys::Hostname;
use Apache::TS::Config::Records;
sub writeFile ($$) { 
  my( $outFile, $content ) = @_;
  print STDERR "Writing " . length( $content ) . " bytes to $outFile\n"; 
  open( FILE, ">" . shift ); print FILE shift; close FILE; 
}
 
#
# Set up context.  The O/P dir can be specified but defaults to /etc/trafficserver
#
 
my $ATS  = "/etc/trafficserver";
my $outDir = ( $#ARGV > -1 ) ? $ARGV[0] : $ATS;
 
die "This procudure must be run from root to write to /etc\n" if $UID > 0 and $outDir =~ m!^/etc!;
my $publicServer = ( hostname == 'ooowikivm' ) ? 'ooowikiv.home' : 'wiki-ooo.apache.org';  
my $wikiServer   = 'localhost';
 
#
# Add mapping rules to remap.conf
#
my $remap = qx(cat $ATS/remap.config.default) . 
              "map          http://$publicServer/ http://$wikiServer/\n" .
              "reverse_map  http://$wikiServer/ http://$publicServer/\n";
writeFile "$outDir/remap.config", $remap;
 
#
# Create storage.conf
#
my( $rawDevice, $diskSize ) = ( "/dev/raw/raw1", 2*1024*1024*1024 );  # 2Gb
writeFile "$outDir/storage.config", "$rawDevice $diskSize\n";
 
my $cfg = new Apache::TS::Config::Records(file => "$ATS/records.config.default");
 
$cfg->set( conf => "proxy.config.exec_thread.autoconfig",                   val => "0"     );
$cfg->set( conf => "proxy.config.exec_thread.limit",                        val => "2"     );
 
$cfg->set( conf => "proxy.config.reverse_proxy.enabled",                    val => "1"     );
 
$cfg->set( conf => "proxy.config.cache.ram_cache.size",                     val => "64M"   );
$cfg->set( conf => "proxy.config.cache.ram_cache_cutoff",                   val => "512K"  );
 
$cfg->set( conf => "proxy.config.cache.ram_cache.compress",                 val => "2"     );
$cfg->set( conf => "proxy.config.cache.threads_per_disk",                   val => "4"     );
 
$cfg->set( conf => "proxy.config.url_remap.remap_required",                 val => "1"     );
$cfg->set( conf => "proxy.config.url_remap.pristine_host_hdr",              val => "0"     );
 
$cfg->set( conf => "proxy.config.http.server_port",	                        val => "80"    );
$cfg->set( conf => "proxy.config.http.insert_response_via_str",             val => "1"     );
$cfg->set( conf => "proxy.config.http.accept_no_activity_timeout",          val => "30"    );
$cfg->set( conf => "proxy.config.http.keep_alive_no_activity_timeout_out",  val => "5"     );
$cfg->set( conf => "proxy.config.http.negative_caching_enabled",            val => "1"     );
$cfg->set( conf => "proxy.config.http.negative_caching_lifetime",           val => "240"   );
$cfg->set( conf => "proxy.config.http.normalize_ae_gzip",                   val => "1"     );
$cfg->set( conf => "proxy.config.http.server_max_connections",              val =>"100" );
 
$cfg->set( conf => "proxy.config.dns.search_default_domains",               val => "0"     );
 
$cfg->set( conf => "proxy.config.hostdb.size",                              val => "1000" );
$cfg->set( conf => "proxy.config.hostdb.storage_size",                      val => "2M"  );
 
$cfg->set( conf => "proxy.config.ssl.enabled",                              val => "0"     );
$cfg->set( conf => "proxy.config.ssl.number.threads",                       val => "0"     );
 
$cfg->set( conf => "proxy.config.http_ui_enabled",                          val => "3"  );
 
#$cfg->append( line => "CONFIG proxy.config.http.enable_http_info INT 1" );
#$cfg->set( conf => "proxy.config.mlock_enabled", val => "2" ); 
$cfg->write( file => "$outDir/records.config" );

Configuring MediaWiki

Since Traffic Server is captures the end-user browser requests and forwards those which requre processing by Apache through the localhost loopback connector, Apache will alsways receive "127.0.0.1" as the direct remote address. However, as Traffic Server forwards requests to Apache, it is configured to add the "X-Forwarded-For" header so that the remote address from the outside world is preserved. MediaWiki must be configured to use the "X-Forwarded-For" header in order to correctly display user addresses in Special:RecentChanges.

The required configuration for Traffic Server is essentially the same as for Squid, with the following config assignments in LocalSettings.php:

$wgUseSquid = true;
$wgSquidServers = array('127.0.0.1');
// $wgInternalServer = '';           // Internal server name as known to Squid. NOT SET.
// $wgMaxSquidPurgeTitles = 0        // Maximum no of pages to purge in one client operation. NOT SET.
// $wgSquidMaxage =  Cache timeout for the squid.
$wgUseXVO = true;                    // Send X-Vary-Options header for better caching.
$wgDisableCounters = true;           // Disable collection of Page counters
$wgShowIPinHeader = false;           // Disable display of IP for guests as this frustrates caching

These settings serve two main purposes:

• If a request is received from the Traffic Server cache server, the MediaWiki logs need to display the IP address of the user, not that of Traffic Server. A Special:RecentChanges in which every edit is reported as '127.0.0.1' isn't meaningful. Listing this address in $wgSquidServers lets the application know that the user IP address should be obtained from the 'x-forwarded-for' header.
• Whenever a page or file is modified on the wiki, MediaWiki must be configured to send Purge notification to any caches which serve its content. $wgSquidServers contains the list of such servers. (The name is misleading. Squid was the first cache supported by MediaWiki.)

Note that the configuration is already tuned to support PHP APC acceration for both MediaWiki code, and metadata caching.

Outstanding issues

• Logging and Page Stats. Most inbound request will be handled by the Traffic Server cache, so the internal stats collected by MediaWiki will only reflect cache misses. We need to think about how we handle logfile analysis and stats in general. I have turned off page counters as these will only reflect cache misses in future.

• Decision to retain a version MediaWiki 1.15 baseline. For MediaWiki v1.16.x and later, internationalisation can add a material D/B load, For this and other schema changes, we've decided to stick with the last stable MW 1.15.x version (1.15.6) as the S/W baseline

Apache configuration

The Apache server is configured to listen on the standard port at the localhost IP, and accepts all requests from Traffic Server:

Listen 127.0.0.1:80

The Apache web server default logging format would only list 127.0.0.1 as the connecting address. Hence and extra "cached" logging option is enabled^[4], and this captures the originating browser's address by using the "x-forwarded-for" header passed by Traffic Server.

LogFormat "%{X-Forwarded-for}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" cached
CustomLog /var/log/apache2/access.log cached

See also

• Cache
• Squid caching

References

1. ↑ http://trafficserver.apache.org/
2. ↑ http://trafficserver.apache.org/docs/v2/admin/reverse.htm
3. ↑ http://trafficserver.apache.org/docs/v2/admin/configure.htm
4. ↑ http://httpd.apache.org/docs-2.2/mod/mod_log_config.html

Copyright © 2011 The Apache Software Foundation. Licensed under the Apache License, Version 2.0.
Apache Traffic Server, Apache, the Apache Traffic Server logo, and the Apache feather logo are trademarks of The Apache Software Foundation.