System Administration Team

We are responsible for HUMBUG's various internet services.

Contact

Email: <sasig AT humbug DOT org DOT au>

For all member requests, please login to Trac as user "guest" with password "guest" and log a ticket.

Please note that tickets and email to sasig are archived and publicly accessible.

If you prefer your issue to be dealt with privately email the club librarian at <librarian AT humbug DOT org DOT au>.

If your request requires secret information like passwords please log a Trac ticket without the secret information, and send the secret information on to the librarian in a separate email. Please include your Trac ticket number in the subject.

Members

System Administrators:

Brad Marshall
Greg Black
James Iseppi
Mark Suter
Mark Ellem
Matthew Franklin
Raymond Smith
Russell Stuart
Robert Brockway
Stephen Thomas

Meetings

These meetings will from the 1/5/2010 will occur straight after an Exec meeting.

SysAdmin Minutes

Guide Lines

There are a few of us, and we are not in face to face contact. I'd like to suggest the following guideline for any change we might make:

The usual rules apply: think before you type, if you don't know check
When making changes:
1. Make a record of your change:
  1. Version Controlled Files: check in message should include a brief note on why you made the change.
  2. Other Files: include a comment near the change with YYYY-MM-DD format and your initials and a brief note why you made the change
2. Email the sysadmin@ list when you are done and update the trouble ticket/respond to the requestors email

The Backup System

Backups are implemented using the Rdiff-Image tools. As they aren't part of Debian Stable there are also in the Humbug Debian Repository. Backups are run hourly by cron(8) from /etc/cron.d/rdiff-image. The most recent backup is stored on excalibur itself and can be downloaded using http from the web site. Amazon S3 stores backups going back 6 months.

Read the rdiff-image-cron(8) man page to gain an understanding of how Rdiff-Image structures its backups. Underneath, the backup is just a tar image of the entire VM. That tar image is split into two. The main backup has all sensitive data stripped out, and all passwords replaced with x. Thus it is sanitised but remains bootable. The second backup (called the secret backup) has the original versions of the sanitised files, so that when restored over the main backup a faithful copy of the VM's file system is created. Both backups can be downloaded by anybody, but the secret backup is encrypted with several gpg keys.

Downloading and Booting Backups

We used to store the instructions for downloading and booting from the backups in the wiki. However, that's not much use if it's a live restore required as a result of excalibur disappearing for some reason. Since we don't want to risk having two sets of instructions getting out of sync, all the instructions are now kept in a Mercurial repository on excalibur and all Sysadmins are expected to obtain the instructions and to follow at least the part under the heading Preliminary Steps - Do This Now in the file README that will be part of the repository that is cloned in the command below:

    hg clone ssh://excalibur.humbug.org.au:24//etc/rdiff-image

If that README needs to be modified, make sure you both update the excalibur Mercurial repository and send a notice to the sasig mailing list so that the other Sysadmins will know they need to update their clone of the repository. The basic Mercurial commands needed for the above steps are discussed in the section on DNS below.

Maintenance of the backup system

Finally, the Amazon S3 costs are kept to a minimum by keep the differences between successive backups small. If they are growing more quickly than you expect, trying using rdiff-image-tarutil(1) to find out why. How do to that is explained in its man page. In fact, lots of important things not described here are in the various Rdiff-Image man pages. Read them.

Common Tasks

Even if it is obvious, we are not all doing this stuff every day. Any common tasks could go here

DNS

zones are stored in mecurial dvcs.

Get a clone of the repo if you have not already done so:

      hg clone ssh://excalibur.humbug.org.au:24//etc/bind/hg/pri

Update your repo:

      hg pull
      hg update

Edit the files in your repository
Update Excalibur:

    hg commit -m "A Meaningful Log Message"
    hg push
    ssh -p24 excalibur.humbug.org.au
    cd /etc/bind/hg/pri   # or whatever location the repo is
    hg update
    sudo ./install.sh

@humbug.org.au alias

We use postfix.

Edit /etc/postfix/virtual/all and make the appropriate change
run postmap /etc/postfix/virtual/all
restart post fix /etc/init.d/postfix restart

Mailman and @humbug.org.au aliases

When subscribing an @humbug.org.au alias and the corresponding destination address you should set the "no email" option on the destination address not the alias.

The reason is that doing it the other way around (disable delivery for the alias, enable for the destination) may disable delivery for both addresses. At least one user (Matthew Franklin) has reported this problem, while at least one user (Raymond Smith) has it working with alias enabled and destination disabled.

planet.humbug.org.au

To add/remove/update blogs edit /srv/http/planet.humbug.org.au/data/config.ini . The format is straightforward. Planet should pick up the changes on its hourly runs.