New World Backup System

Overview

• The NW (New World) Backup System is built around rsnapshot(1) as its underlying technology
  • rsnapshot is a remote filesystem snapshot utility that uses rsync(1);
  • rsync is a fast and versatile, remote (and local) file and directory copying tool that

• Handles local system snapshots directly, and
• Handles remote system snapshots using ssh(1).

• Typically, rsnapshot is set up to store snapshot copies of source directories in an archive filesystem.

Periodically (typically each day) rsnapshot makes a new snapshot of its source directories:

1. Copying to the new snapshot only those files that are new, or which have changed, since the last copy was made;
2. Retaining links in the new snapshot to those files in the previous snapshot that remain unchanged.

• The CSE backup system is run on the New World AWS machine:

nw-syd-backup1.

• The scripts, logs, and archives associated with NW backups are generally found in the directory:

nw-syd-backup1:/export/nw-syd-backup1/1/backups/ (aka: ~backups)

Operation

1. A separate rsnapshot archive is kept of each CSE directory belonging to each CSE user.

   These archives are accessible via the directory:

   ~backups/users/$LASTTWO/$user[.n]/

   where:

   • $LASTTWO is the last two characters of the user's CSE username,
   • $user is the name of the CSE user whose directory is stored in the archive.
   • [.n] is a unique numeric suffix assigned to each of the user's different CSE directories (if the user has more than one).

2. The copy (or snapshot) of the user's directory that was made N days ago, is found in the user's archive directory $user[.n]/, under the directory named: daily.N/.

   • There are currently a maximum of 30 daily snapshots of the user's directory stored in each archive, one for each day since the snapshot it contains was made.
   • These daily.N/ directories are renamed (ie: shifted up) each day that they age, until finally the copy made 30 days ago (named daily.30/) is deleted.

3. The path: basename/export/hostname/partition/basename/ is always inserted by rsnapshot between each directory daily.N/ and the actual snapshot/copy of the user's directory.

   • There does not seem to be a way of avoiding this pathname being inserted by rsnapshot given the particular organisation of user archives we use in the NW backups.
   • basename will usually be the same as the username.
   • Although this pathname may be used to uniquely identify the archive's origin, the file: $user[.n]/.bu_source is more conveniently used to do this.

4. The same access (ie: ownership and permissions) is assigned to each file and directory in the user's backup archive as were assigned to each file and directory in their original CSE (home) directory.

   Note however:

   • The filesystems storing the CSE archives are mounted read only outside of the backup server,
   • This means that no user can change the contents or access permissions of any file or directory in their archive, even if they seem to have permissions to do so.
   • This means that it may be possible for users to not be allowed access to files or directories in their archive, if they were originally copied with the wrong access permissions.

   In this case, the users will need to get the help of the system staff (root) to access these files/directories.

5. Once a user's CSE account has expired, and/or the user's CSE directory has been removed from its host server, the user's archive of their directory will also be moved aside

   from: ~backups/users/$LASTTWO/$user[.n]/

   to: ~backups/users/$LASTTWO/.deleted/$user[.n]/.

   However:

   • The daily.N/ directories will continue to be shifted up daily until they have all been removed.
   • If the user's CSE directory is restored to the CSE host sever before all 'daily.N/' dirs have been removed, then the remaining daily.[N..M]/ directories will be renamed daily[0..M-N], and made available to the user once more.

Locations

CSE's backup server

nw-syd-backup1@cse.unsw.edu.au

~backups

nw-syd-backup1:/export/nw-syd-backup1/1/backup/

~backups/users/$LASTTWO/$user[.n]/

This is the location of $user's rsnapshot archives, such that:

• $LASTTWO is the last two characters of the user's CSE username,
• $user is the name of the CSE user whose directory is stored in the archive.
• If the user has been allocated more than one CSE directory, then [.n] is a unique numeric suffix assigned to all but one of the user's CSE directories.

The archive without the numeric suffix should always be the archive that stores the user's main home directory.

• $user[.n] is actually a soft link to the actual storage location of the backup archive: backups/disks/$mountpoint/$LASTTWO/$user[.n]

~backups/users/$LASTTWO/$user[.n]/.bu_source

This file identifies the source of the actual user's directory that is stored in this rsnapshot archive. It will mpst likely be of the form: /import/bach/1/username

~backups/users/$LASTTWO/.deleted/$user[.n]/

If $user's source directory, as identified by the file: $user[.n]/.bu_source, no longer exists on the source filesystem, then the user's archive is moved into this .deleted/ directory.

~backups/disks/$mountpoint/$LASTTWO/$user[.n]/

This is the actual storage location of the backup archive for $user[.n].

There may be many different $mountpoint each of which represents a separate (possibly virtual) disk storage filesystem on which user's archives may be stored. This provides a way of managing disk space allocation.

The directory: ~backups/users/$LASTTWO/$user[.n]/ is actually a symlink to the actual directory in ~backups/disks/$mountpoint/$LASTTWO/$user[.n] that has been allocated to this user's backup archive.

~backups/var/

The location of various files containing (user/dir) data.

These files group (user/dir) data into different classifications which are used to determine which (user/dir)s are to be snapshotted, and how their archives are to be managed.

~backups/lib/

The location of most of the configuration files used by the NW backup system.

~backups/bin/

The location of most of the scripts used by the backup system, including the main backup script itself (described in the next section).

Scripts

runbackup.sh

This is the main script tying all the other scripts together that relate to backups.

Location

~backups/bin/runbackups.sh

Called by

nw-syd-backup1:/etc/cron.d/rsnapshot

Calls

logrotate(1)

This specifically rotates those logs in ~backups/logs/ that have anything to do with new backups.

logrotate is passed the specfile: ~backups/lib/logrotate.conf

backup -q run -p 5

This is the main backup command that updates the backup archives. See: backup and run.

truncate.sh

This looks for and truncates excessively long archive directories that may have caused the backups of these archives to fail. See: truncate.sh.

rate.pl

This produces summaries of backup rates (seconds per user). See: rate.pl

This script is called twice:

1. With '-s' to append a rate summary to ~backups/logs/rate.s.log
2. With '-r' to append a progressive rate summary to ~backups/logs/rate.r.log

backupstat

This script attempts to gauge the success of the last NW Backups (run by backup above).

It does this by counting the number of archives with new/changed files in them. See: backupstat.

This script is called twice:

1. With '-f' to force the script to find and count the new/changed files in each archive.
2. With '-n -q' to append the summary of the last find to ~backups/logs/rate.r.log

Log

~backups/logs/runbackups.log

Records the last time runbackups.sh was run, and which and when the various scripts were run.

backup

This is the main script that contains most of the bespoke commands used by the backup system.

Location

~backups/bin/backup

Called by

runbackups.sh

Many of the commands within backup can be run individually by passing them as arguments to backup, along with their desired options.

The backup -h option produces help documentation describing backup and its constituent commands. This documentation is duplicated below, with additional details where this might be helpful.

Manual Page

Usage

backup [-h] [-l log] [-n] [-q] command [commandargs]

Function

This script deals with CSE rsnapshot backups made for any CSE user who owns a home, or other directory, that is hosted on a CSE file server.

This script should usually be run on the CSE backup server: nw-syd-backup1.

Options

-h [ command | topic ]

Print help for the specific command or topic passed. If no command or topic is passed, print this general help and exit.

Topics and commands are summarised below.

-l logfile

Log messages to logfile (Default: ~backups/log/)

-n

Do not make any changes - just report what would be done.

-q

Do not reproduce messages on STDERR.

command [command_args]

Run the backup command with its optional command arguments.

• Backup commands are summarised below.
• Full descriptions of backup commands and their args may be produced by running:backups -h command. They are also included in what follows, under their own subsections.

Topics

general

Usage: backups [-h] [-l log] [-n] [-q] command [commandargs]

locations

Locations used by the CSE Backups System

overview

Overview of CSE Backup System

Commands

diff

User related diff.

fixprimary

Fix primary home archive names if necessary.

mkspec

Prepare backup archive directories and specfiles for users.

movearch

Copy backup archives to another backup filesystem.

movesource

Change the recorded source directory for a user's backup archive.

purge

Shifts and/or deletes archives of expired users.

rename

Renames archive(s) of a CSE user whose username has changed.

report

Summarise logs to report on the last backups.

resurrect

Resurrect a user's inactive backup archive from deleted archives.

run

Create and/or update CSE New World user backup archives.

run

This is the main backup command, which actually runs the rsnapshot backups for each CSE user.

Usage

backup [-genopts] run [-m MAXSPEC] [-N[123]] [-n] [-p MAXPROC] [-s SCRIPT] [-u user]

Function

Create and/or update CSE New World user backup archives.

1. Create lists of all CSE (user/directories) in ~backups/var/.

   • Classify these directory sources
   • Fix primary homes if necessary (See: fixprimary command below)

2. Create specfiles
   1. Ensure a backup archive exists for each (user/dir).

      (archives in $BU_USERS/)

   2. Create rsnapshot specfiles for each (user/dir).

      (specfiles in /var/tmp/backups/)

3. Use xargs to run consecutive invokations of SCRIPT, passing a different specfile to each process each time.

   (See '-s SCRIPT' option below)

Options

-D

Run an xargs process for each physical disk storing archives.

This attempts to evenly distribute simultaneous disk activity across all physical disks.

Default: Only run one xargs process, and ignore details of physical storage disks.

-m MAXSPEC

Each xargs process will call SCRIPT no more than MAXSPEC times.

Default: Call SCRIPT until all specfiles are processed.

-N1

Do NOT (re)create user dir lists. (ie: Do not run Function step 1)

-N2

Do NOT (re)create rsnapshot dirs or specfiles. (ie: Do not run Function steps 2.1, 2.2)

-N3

Do NOT use xargs to call SCRIPT at all. (ie: Do not run Function step 3).

-n

Equivalent to -N3 or '-m 0'.

-p MAXPROC

Each xargs runs MAXPROC simultaneous process sequences of SCRIPT.

Default:

• With '-D': 1;
• Without '-D': 6

-s SCRIPT

xargs consecutively invokes this SCRIPT (in step 3),

passing SCRIPT another specfile each time.

Default SCRIPT - ~backups/bin/run_rsnapshot:

1. Passes the specfile to rsnapshot with the required options.
2. Removes the specfile after rsnapshot has run.

Note: SCRIPT may be any program. It need not run rsnapshot, nor must it do anything with the specfile passed.

-u user

Create the specfiles for just this user (unless '-N2'),

Run SCRIPT passing just user's specfiles (unless '-n')

(Default: Create and run specfiles of all CSE users).

mkspec

Usage

backup [-genopts] mkspec [-D] [-nfs] [file]

Function

Prepare backup archive directories and specfiles for users specified in file.

• This is the function called by the run command to perform its Function steps 2.1 and 2.2.
• Read the input file of format: directory user host.

For each (user/dir) for which host is defined:

1. Find or create an rsnapshot archive for (user/dir) in ~backups/users/$LASTTWO/$user[.n]/.
2. Create individualised rsnapshot specfiles for each user/dir. (By default in /var/tmp/backups/)

If host is not defined, but an rsnapshot archive has been found for this (user/dir), then move the archive to ~backups/users/$LASTTWO/.deleted/$user[.n].

If host is not defined, but no rsnapshot archive is found, then silently ignore the (user/dir).

Options

-D

Create the specfiles in /var/tmp/backups/disk.N/ depending on the actual disk partition on which the archive is stored. This allows the run command to be run with the -D option.

-nfs

The rsnapshot specfile that is created, will specify that the directory should be accessed over NFS, rather than via an SSH connection to host (which is what is specified by default).

file

This is the input file of format: directory user [host].

If host is present then the directory exists on host.

(Default file: ~backups/var/ssh_dirs)

report

Usage

~backups/bin/backup [-genopts] report [-d date]

Function

Summarise logs to report on the last backups.

Options

-d date

Poduce a report on all backups run on this date.

Date is specified as yyyy.mm.dd

NOT YET IMPLIMENTED

diff

Usage

~backups/bin/backup [-genopts] diff user

Function

User related diff.

*NOT YET IMPLIMENTED*

movearch

Usage

~backups/bin/backup movearch [archname tofs | -f file]

Function

The backup archive archname is copied from its current archive filesystem to another.

The backup archive: ~backups/users/$LASTTWO/$archname

which is a soft link to its actual location at: ~backups/disks/N/$LASTTWO/$archname

is copied to a new location at: ~backups/disks/M/$LASTTWO/$archname.

If the copy is successfull, the soft link is adjusted to the new location.

Options

archname

The name of the specific archive to be moved

This is usually of the form: 'username[.[0-9]]'

tofs

Full pathname of the destination filesystem (eg: /export/nw-syd-backup1/1/backups/disks/3)

-f file

Read file of format: archname tofs, and copy and move each archive to tofs.

If file is -, then the script will read from STDIN.

Default: No options or args => Read from STDIN. (ie: -f - )

Note

This command does not change the recorded source of the archive (ie: the location of the user's actual home directory).

If you have moved the source of the archive from one home directory server/filesystem to another, then you should use the movesource command instead.

movesource

Usage

~backups/bin/backup movesource user old_source_pathname new_source_pathname

Function

Change the source_pathname recorded for a user's backup archive.

Details

The full source_pathname of each backup archive (ie: the full pathname of the user's actual home or other directory) is recorded in two places:

1. The file: ~backups/users/$LASTTWO/$username[.n]/.bu_source

   This file is written and checked by backup.

2. In each archive's retention level pathname:

   ~backups/users/$LASTTWO/$username[.n]/daily.N/$username/$source_pathname/

   The actual directory pathname matching source_pathname is written and checked by rsnapshot.

Use

Run this command when a user's home (or other directory) has had its pathname changed in any way, either because:

1. Some directories in the pathname were renamed (but the contents were not physically relocated), or
2. The directory and its contents were relocated/copied from one disk server's file system to another, resulting in a pathname change.

Example

~backups/bin/backup movesource zain /import/kamen/1/zain /import/glass/A/zain

Note

• This command only changes the archive source records in Details (1) and (2) above.

The actual relocation/renaming necessitating the movesource are expected to be done separately elsewhere.

• If $username has also changed, and not just the source_pathname, then you should also use the rename command.

rename

Usage

backup rename oldusername newusername

Function

Renames archive(s) of a CSE user whose username has changed.

Details

In general, backup archives belonging to username will include the following access pathnames:

nw-syd-backup1:/export/nw-syd-backup1/1/backups/users/$LASTTWO/$username[.n]/daily.N/$username/$source_pathname/

This command changes all occurences of $username in such access pathnames from oldusername to newusername, also changing $LASTTWO where necessary.

Note

This command will not change any further occurences of $username that may occur within $source_pathname. If $source_pathname has changed in any way, then you should also use the movesource command.

resurrect

Usage

backup resurrect user [sourcedir]

Function

Resurrect a user's inactive backup archive from deleted archives.

If sourcedir is passed, only resurrect the user's inactive backup archive coming from sourcedir, otherwise resurrect all inactive backup archives belonging to user.

Details

1. Active CSE users have backup archives kept in: ~backups/users/$LASTTWO/username[.n]/
2. Expired CSE users have their backup archives made inactive and moved to: ~backups/users/$LASTTWO/.deleted/username[.n]/

If an expired CSE user has their account reactivated, then this command reactivates their backup archive by moving it from (2) to (1).

Note

• This command only resurrects the backup archive (if it exists), so that the user's home directory(s) may be backed up once more.

It does not restore the user's original home directory from the backup archive. Run restore to do this.

• The source of the resurrected backup archives are assumed to stay unchanged.

If the user's original directory(s) is restored into different file systems/sourcedirs, in addition to resurrecting the user's original archive, you may need to use movesource and/or rename.

See Also

purge - for what happens to deleted archives.

movesource - for moving archive sources.

rename - for changing user names.

restore - for restoring user directory from archives.

purge

Usage

backup [-genopts] purge [-u user] [-m MAX] [-n]

Function

Shifts and/or deletes expired backup archives.

Details

When CSE users are expired and/or their home or other directory has been deleted, the run command moves the associated archives aside

from: ~backups/users/$LASTTWO/user[.n]/

to: ~backups/users/$LASTTWO/.deleted/user[.n]/.

Thereafter, each time this command is run, this command:

1. Shifts up the retention directories of all expired archives;
2. Removes any expired archives that have no retention directories left.

Options

-u user

Just shift and/or purge rsnapshot archives belonging to user.

(Default: Shift and/or purge all user archives in .deleted/ directories)

-m MAX

Only shift and/or purge MAX users.

-n

Do not actually shift or remove any rsnapshot archives. Just list archives that are affected on STDOUT.

-v

verbose listing - include disk usage

fixprimary

Usage

backup fixprimary

Function

Fix primary home archive names if necessary.

For every CSE user with:

• A CSE home and/or other directory, and
• At least one CSE backup archive of these home/directories,

Ensure that the user's primary home directory is stored in their primary backup archive (ie: their primary home directory is not stored in a backup archive with a .N suffix).

This command uses renamearch to rename archives if necessary.

truncate.sh

This script truncates excessively long directories from archives that can cause rsnapshot(1) to fail.

Location

~backups/bin/truncate.sh

Called by

runbackup.sh

Manual Page

Usage

truncate.sh [-n] [-f] [-d depth] [-l outfile] [infile]

Function

This script truncates excessively long directories from archives that can cause rsnapshot(1) to fail.

Details

rsnapshot, the basis of CSE's New World Backup system, relies on the command cp to copy/link:

From the user's last backup directory archive (renamed daily.1/),

To the user's new backup directory archive (called daily.0/).

However, if daily.1/ contains any directories that are too deeply nested, then the recursive copy/link fails, and the rsnapshot of that archive fails after it records an error message in its logfile.

This script is intended to be run after rsnapshot is run. It:

1. Reads rsnapshot logfile infile, and looks for error messages that may indicate problems copying deeply nested directories.
2. Adds these potentially problematic archives to the list of previous archives that needed truncation: ~backups/logs/truncate.users.
3. Searches these archives for directories nested deeper than depth levels.
4. Truncates these directories within their last archive (daily.0) at depth.
5. Logs changes in ~backups/logs/truncate.log
6. Rewrites the archives needing truncation to ~backups/logs/truncate.users.

Note

• This means that no archive will have deeply nested directories, except for those deeply nested directories that have not yet been rsnapshot'ed a second time (requiring the copy and thereby creating the initial error message).
• This script is only necessary if cp keeps this depth related limitation.

Options

-n

Do not make changes or log anything to outfile. Just report what changes would have been made.

(Default: Do not make changes if truncate.users is newer that $outfile).

-f

Force changes to be made and logged to outfile.

Beware: truncate.sh should only be forced to make changes if backups has been run beforehand.

(Default: Make changes if and only if truncate.users if older than $outfile)

-d depth

The minimum depth of the pathnames to be truncated. Default: 200

-l outfile

Output log file. Must be an absolute pathname.

Default: ~backups/logs/truncate.users

-q

Quiet; If this script is making changes (ie: no -n) then touch outfile rather than append timestamp to outfile.

infile

Input file. May be a relative pathname. Default: ~backups/logs/rsnapshot.log

rate.pl

Location

~backups/bin/rate.pl

Called by

runbackup.sh

Usage

rate.pl [-s] [-l [-p pNO] | -r [-n MAX]] [file]

Function

Produce backup runtime stats (seconds per user) from either:

1. backup logs (default or -l), or
2. backup records (-r).

By default, this produces stats for each day from this month, including when the backup finished, how long it ran, the total number of archives processed, and the average number of seconds per user/archive.

Options

-l

Produce stats from backup logs.

This log contains records from this month's backup runs, and the default stats include (secs/user) for each day from this month.

This is the default, and excludes -r.

(Default file: ~backups/logs/backuplog)

-n MAX

Produce a rate calculation every MAX users.

Applies only to stats from rsnapshot logs.

(Default: 1000)

-p PNO

Only produce stats for process PNO.

Applies only to stats from backup logs, which include process id in logs.

-r

Produce stats from run_rsnapshot logs.

This log contains the single day's records for the last run of backups, and the default stats includes (secs/user) stats for every 1000 users.

This excludes -l;

(Default file: ~backups/logs/bu_record)

-s

Summary; Print the last (secs/user) stat only.

[file]

The file of records to be read

'-' reads from STDIN.

backupstat

Location

~backups/bin/backupstat

Called by

runbackup.sh

Usage

backupstat [-N] ( [-f] [-n] [-q] {logfile} | [-u user] )

Function

Attempt to gauge the success of the last New World Backup run.

Report on the number of CSE user archives in:

~backups/users/

that have at least one new (day old/changed) file in their archive:

daily.0/

Note:

• Given the conditions:

1. Backups have failed, and
2. N = Number of archives containing any new/changed files.
3. (1) implies (2) N=0, but
4. (2) N=0 does not imply (1)

• We want to determine (1) but can only measure (2).
• Despite (4), (2) is still a good measurement to make.

Details

This script:

1. For each user, looks for files that are either:
   1. Less than one day old (Default).
   2. Have only one link (-l) - ie: new/changed since last backup.

   Adds user's result to logfile: ~backups/logs/dayold

2. Summarizes the logfile.

By default, step (1) is only performed and the log file recreated if the logfile was not already modified today.

(See also options below to change this behaviour).

Options

-l

Find files with one link (fast, but not a good test)

Default: Find files less than one day old (slower).

-u user

List all match files in user's archive.

Ignore all other following options and logfiles (except -N).

-c

Count all files found (slow).

Default: Quit after first file found - ie: produce 1 or 0 (fast).

-f

Force logfile to be recreated even if it was modified today.

-n

Do not recreate log file, even if it was not modified today.

This just produces the summary report from the existing log file.

This option overrides '-f'.

-q

Quiet - just produce summary.

logfile

Produce a report from logfile(s). Implies '-n'.

Note: logfile(s) may be compressed (.gz).

Default: ~backups/logs/dayold