System Administration Duties

Appointed by the Executive

The Executive must nominate one or more people to look after HUMBUG's computer systems.


The system administration team should submit a brief report to every Executive meeting in the following format:



Email: <sasig AT humbug DOT org DOT au>

For all member requests, please login to Trac as user "guest" with password "guest" and log a ticket.

Please note that tickets and email to sasig are archived and publicly accessible.

The Backup System

Backups are implemented using the Rdiff-Image tools. As they aren't part of Debian Stable there are also in the Humbug Debian Repository. Backups are run hourly by cron(8) from /etc/cron.d/rdiff-image. The most recent backup is stored on excalibur itself and can be downloaded using http from the web site. Amazon S3 stores backups going back 6 months.

Read the rdiff-image-cron(8) man page to gain an understanding of how Rdiff-Image structures its backups. Underneath, the backup is just a tar image of the entire VM. That tar image is split into two. The main backup has all sensitive data stripped out, and all passwords replaced with x. Thus it is sanitised but remains bootable. The second backup (called the secret backup) has the original versions of the sanitised files, so that when restored over the main backup a faithful copy of the VM's file system is created. Both backups can be downloaded by anybody, but the secret backup is encrypted with several gpg keys.

Downloading and Booting Backups

We used to store the instructions for downloading and booting from the backups in the wiki. However, that's not much use if it's a live restore required as a result of excalibur disappearing for some reason.

Since we don't want to risk having two sets of instructions getting out of sync, all the instructions are now kept in a Mercurial repository on excalibur and all Sysadmins are expected to obtain the instructions and to follow at least the part under the heading Preliminary Steps - Do This Now in the file README that will be part of the repository that is cloned in the command below:

    hg clone ssh://

If that README needs to be modified, make sure you both update the excalibur Mercurial repository and send a notice to the sasig mailing list so that the other Sysadmins will know they need to update their clone of the repository. The basic Mercurial commands needed for the above steps are discussed in the section on DNS below.

Maintenance of the backup system

Finally, the Amazon S3 costs are kept to a minimum by keep the differences between successive backups small. If they are growing more quickly than you expect, trying using rdiff-image-tarutil(1) to find out why. How do to that is explained in its man page. In fact, lots of important things not described here are in the various Rdiff-Image man pages. Read them.

Common Tasks

Even if it is obvious, we are not all doing this stuff every day. Any common tasks could go here

Stopping and Starting services

Debian stretch uses systemd as it's init system, but some VPS's don't support it. To get around that use this command to start a service:

     /usr/bin/rdiff-image-boot start-service service-name

service-name can by a systemd service (ie, there is a file called service-name.service), or a SysV init script called /etc/init.d/service-name.


zones are stored in mecurial dvcs.

  1. Get a clone of the repo if you have not already done so:

      hg clone ssh://
  1. Update your repo:

      hg pull
      hg update
  1. Edit the files in your repository
  2. Update Excalibur:

    hg commit -m "A Meaningful Log Message"
    hg push
    ssh -p24
    cd /etc/bind/hg/pri   # or whatever location the repo is
    hg update
    sudo ./ alias

We use postfix.

  1. Edit /etc/postfix/aliases-humbug and make the appropriate change

  2. run sudo postfix reload

Mailman and aliases

When subscribing an alias and the corresponding destination address you should set the "no email" option on the destination address not the alias.

The reason is that doing it the other way around (disable delivery for the alias, enable for the destination) may disable delivery for both addresses. At least one user (Matthew Franklin) has reported this problem, while at least one user (Raymond Smith) has it working with alias enabled and destination disabled.

To add/remove/update blogs edit  /srv/http/ . The format is straightforward. Planet should pick up the changes on its hourly runs.

VM Credentials

The credentials needed to login / control the VM excalibur runs on are attached to this page. Download. It is GPG encrypted to the same people who can decrypt the backup.

Payment Gateway

The payment gateway lives under /srv/http/payments. It is a mercurial repository. There are further instructions in the README.txt file in that directory.

SysAdmin (last edited 2018-04-07 11:36:10 by RussellStuart)