mod_perl has recently been disabled, because there was some suspicion it was what breaking the apache parent.

We shall see…

(for the purposes of undoing the damage)

apache-modconf apache disable mod_perl

Also, some lines were commented out in /etc/apache/conf.d/libhtml-mason-perl, because they were stopping apache from starting. I dont really know what they do, but if it comes round to bite me, I guess we'll find out :)

...
<IfModule !mod_perl.c>
# No mod_perl available, just use CGI
#Action mason_example http://localhost/cgi-bin/mason_example.cgi
#<Directory /var/www/mason_example>
#SetHandler mason_example
#</Directory>
</IfModule>
...

Server version: Apache/1.3.33 (Debian GNU/Linux)
Server built:   Dec 18 2004 11:28:47
Server's Module Magic Number: 19990320:16
Server compiled with....
 -D EAPI
 -D HAVE_MMAP
 -D HAVE_SHMGET
 -D USE_SHMGET_SCOREBOARD
 -D USE_MMAP_FILES
 -D HAVE_FCNTL_SERIALIZED_ACCEPT
 -D HAVE_SYSVSEM_SERIALIZED_ACCEPT
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D DYNAMIC_MODULE_LIMIT=64
 -D HARD_SERVER_LIMIT=4096
 -D HTTPD_ROOT="/usr"
 -D SUEXEC_BIN="/usr/lib/apache/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache.pid"
 -D DEFAULT_SCOREBOARD="/var/run/apache.scoreboard"
 -D DEFAULT_LOCKFILE="/var/run/apache.lock"
 -D DEFAULT_ERRORLOG="/var/log/apache/error.log"
 -D TYPES_CONFIG_FILE="/etc/mime.types"
 -D SERVER_CONFIG_FILE="/etc/apache/httpd.conf"
 -D ACCESS_CONFIG_FILE="/etc/apache/access.conf"
 -D RESOURCE_CONFIG_FILE="/etc/apache/srm.conf"

We've got a silly directory structure, so that we don't need users homedirs mounted on the webserver. Unfortunately this breaks the default debian way of doing things. To get round this we've our own apache package.

apt-get update
mkdir /tmp/apachelol
cd /tmp/apachelol
apt-get build-dep apache
apt-get source apache
cd apache-<version>
vim debian/rules

Then change the config args as follows (note the tardis bit, and the last line):

CONFARGS =      --target=apache --with-layout=Debian \
                --enable-suexec --suexec-caller=www-data \
                --suexec-docroot=/tardis/www --includedir=/$(inc) \
                --without-confadjust --without-execstrip \
                --enable-shared=max --enable-rule=SHARED_CHAIN \
                --enable-module=most --enable-module=status \
                --enable-module=auth_digest --enable-module=log_referer \
                --enable-module=log_agent --enable-module=auth_db \
                $(EXTRA_CONFARGS) \
                --activate-module=src/modules/extra/mod_macro.c \
                --suexec-userdir=cgi-bin

Then:

:wq
tardis-buildpkg publish all
tardis-buildrep
apt-get update
apt-get upgrade

We kept seeing crashes where the master apache process would die. The children would stay around answering requests for a while afterwards, making diagnosis more difficult. From the logs, it would appear to be a problem with sysvsem mutexes and the log rollover. We would get

[Fri Feb  3 06:34:11 2006] [notice] SIGUSR1 received.  Doing graceful restart
accept_mutex_on: Identifier removed

before the rollover, and

[Fri Feb  3 06:34:14 2006] [error] (2)No such file or directory: mod_mime_magic: can't read magic file /etc/apache/share/magic
[Fri Feb  3 06:34:14 2006] [notice] Apache/1.3.33 configured -- resuming normal operations
[Fri Feb  3 06:34:14 2006] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache/suexec)
[Fri Feb  3 06:34:14 2006] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Fri Feb  3 06:34:14 2006] [alert] Child 15864 returned a Fatal error... \nApache is exiting!

afterwards. Googling showed that lots of people have had this problem in the past and noone has bothered finding the bug. Switching to fcntl mutexes appears to have stopped this.