New World Backup System

Overview

Overall Brief

The following was the original overall brief for the CSE New World (NW) Backup System :

1. The Backup system should make a separate archive for every CSE directory that "belongs" to to every CSE user.
2. Each archive should be easily accessible by the user whose directory is being archived.
3. Each separate archive should be independent of each other.
4. Access to a user's archive directories and files should be protected by access permissions at least as strong as those used to protect the user's original source directories and files.
5. New CSE users and/or directories should be able to be added to the backup system, and existing users and/or directories removed from it, without affecting any of the other existing user's archives.
6. A series of daily snapshots should be made of each user's directory, which may subsequently by summarised and stored in a series of weekly, monthly, and yearly archives.

Current Configuration

In the Current configuration, there are progressively:

• 7 daily archives
• 4 weekly archives
• 12 monthly archives
• 1 yearly archive

Technology

The NW Backup System is built around rsnapshot(1) as its underlying technology

• rsnapshot is a remote filesystem snapshot utility that uses rsync(1);
• rsync is a fast and versatile, remote (and local) file and directory copying tool that

• Handles local system snapshots directly, and
• Handles remote system snapshots using ssh(1).

• Typically, rsnapshot is set up to store snapshot copies of source directories in an archive filesystem.

Periodically (typically each day) rsnapshot uses rsync to make a new snapshot of each of its source directories:

1. Copying to the new snapshot archive only those files that are new, or which have changed, since the last copy was made;
2. Retaining links in the new snapshot archive to those files in the previous archive that remain unchanged.

• The CSE backup system is run on the New World AWS backup machine:

nw-syd-backup1.

• The scripts, logs, and archives associated with NW backups are generally stored in the directory on the backup machine:

nw-syd-backup1:/export/nw-syd-backup1/1/backups/ (aka: ~backups)

Operation

Archive Directories

Each CSE directory on a CSE fileserver belonging to a CSE user is backed up and stored in the user's separate rsnapshot Archive Directory located at:

~backups/users/$LASTTWO/$user[.n]/

where:

• $LASTTWO is the last two characters of the user's CSE username,
• $user is the name of the CSE user whose directory is stored in the archive.
• [.n] is a unique numeric suffix assigned to each of the user's different CSE directories (if the user has more than one).

Each archive directory in ~backups/users/ is actually a soft-link to the actual archive directory that stores the archive within the disks mounted on ~backups/disks/[2..5]/.

eg:

# cd /import/nw-syd-backup1/1/backups
# ls -l users/in/zain
lrwxrwxrwx 1 root root 48 Sep  5  2021 users/in/zain -> /import/nw-syd-backup1/1/backups/disks/6/in/zain

This indirection allows the user archives to be distributed evenly (and randomly) over the set of disks allocated to backup archive storage. It also means that the location of the backup storage of the user's archive is completely independent of the location of the actual fileserver storage of the user's source directory.

Identity file

The file: $user[.n]/.bu_source in the archive directory is used to identify the original source (somewhere on one of CSE home fileservers) of the user's CSE directory that is being backed up in the archive directory. If the user's source directory is moved from one home directory server to another (for whatever reason), then the contents of this identity file will need to be changed.

Retention Directories

Copies (or snapshots) are made of the user's directory and kept in various Retention Directories within each user's Archive Directory. These retention directories are named:

1. daily.0/ .. daily.6/
2. weekly.0/ .. weekly.3/
3. monthly.0/ .. monthly.11/
4. yearly.0/

• In general, each retention directory indicates how long ago the snapshot archive that it contains was made. For example daily.3/ contains a copy of the user's directory that was made 4 days ago, whereas weekly.2/ will contain a copy of the user's directory made 3 weeks ago.
• Each retention directory contains a copy of all the files and directories that existed in the user's original source directory at the time that the archive was made. If a file has not changed since the previous snapshot archive was taken, then the file in one archive will be a hard link to the identical file in the previous archive. In this way it is only necessary to take a complete and separate copy of any file if it did not already exist when the previous archive was made. Avoiding unnecessary duplications of identical files saves a lot of space in the archives, but depends critically on maintaining the integrity of the archive filesystem.
• The daily.N/ directories are shifted up to daily.N+1/ each day that they age, while daily.6 is either shifted up into weekly.0 (once a week), or else it is deleted.
• Similar progressive shifts and/or deletions occur for each other set of retention directories.

Access

The same access (ie: ownership and permissions) is assigned to each file and directory in the user's backup archive as were assigned to each file and directory in their original CSE (home) directory.

Note:

• The filesystems storing the CSE archives will only ever be mounted read-only on any machine on which users will be able to login.
• This means that no user will be able to change the contents or access permissions of any file or directory in their archive, even if they seem to have permissions to do so.
• This also means that it is possible for users to be disallowed access to files or directories in their archive, if they were originally set up (and copied) with such restrictive access permissions.

In this case, the users will need to get the help of the system staff (root) to access these files/directories.

Account Expiry

Once a user's CSE account has expired, and/or the user's CSE directory has been removed from its host server, the user's archive of their directory will also be moved aside

from: ~backups/users/$LASTTWO/$user[.n]/

to: ~backups/users/$LASTTWO/.deleted/$user[.n]/.

Note:

• In both cases $user[.n] remains a soft-link pointing to the same actual archive directory storing the the user's archive.
• The retention directories within the user's archive will continue to be shifted up as appropriate until they have all been removed.
• If the user's account is reactivated and the user's CSE directory is restored to the CSE host sever before all retention dirs have been removed, then the youngest retention directory is renamed daily.0/, and the remaining directories retained as is. All retention directories are subsequently shifted up when and as necessary.

Locations

CSE's backup server

nw-syd-backup1@cse.unsw.edu.au

~backups

nw-syd-backup1:/export/nw-syd-backup1/1/backup/

~backups/users/$LASTTWO/$user[.n]/

This is the location of $user's rsnapshot archives, such that:

• $LASTTWO is the last two characters of the user's CSE username,
• $user is the name of the CSE user whose directory is stored in the archive.
• If the user has been allocated more than one CSE directory, then [.n] is a unique numeric suffix assigned to all but one of the user's CSE directories.

The archive without the numeric suffix should always be the archive that stores the user's main home directory.

• $user[.n] is actually a soft link to the actual storage location of the backup archive: backups/disks/$mountpoint/$LASTTWO/$user[.n]

~backups/users/$LASTTWO/$user[.n]/.bu_source

This file identifies the source of the actual user's directory that is stored in this rsnapshot archive. It will mpst likely be of the form: /import/bach/1/username

~backups/users/$LASTTWO/.deleted/$user[.n]/

If $user's source directory, as identified by the file: $user[.n]/.bu_source, no longer exists on the source filesystem, then the user's archive is moved into this .deleted/ directory.

~backups/disks/$mountpoint/$LASTTWO/$user[.n]/

This is the actual storage location of the backup archive for $user[.n].

There may be many different $mountpoint each of which represents a separate (possibly virtual) disk storage filesystem on which user's archives may be stored. This provides a way of managing disk space allocation.

The directory: ~backups/users/$LASTTWO/$user[.n]/ is actually a symlink to the actual directory in ~backups/disks/$mountpoint/$LASTTWO/$user[.n] that has been allocated to this user's backup archive.

~backups/var/

The location of various files containing (user/dir) data.

These files group (user/dir) data into different classifications which are used to determine which (user/dir)s are to be snapshotted, and how their archives are to be managed.

~backups/lib/

The location of most of the configuration files used by the NW backup system.

~backups/bin/

The location of most of the scripts used by the backup system, including the main backup script itself (described in the next section).

Scripts

runbackup.sh

This is the main script tying all the other scripts together that relate to backups.

This script records its process id in the file:~backups/var/runbackups_pid, which this script checks to ensure that no previous process is still running. If it is, then this script sleeps for 10 minute intervals for a maximum of 3 hours, waiting for the process to finish. If the previous process does not finish within 3 hours, then this script terminates with an error.

Location

~backups/bin/runbackups.sh

Called by

Cron entry: nw-syd-backup1:/etc/cron.d/rsnapshot

Calls

logrotate(1)

This specifically rotates those logs in ~backups/logs/ that have anything to do with new backups.

logrotate is passed the specfile: ~backups/lib/logrotate.conf

backup -q run

This is the main backup command that updates the backup archives. See: backup and run.

Be aware that it is possible to run backup run independently of, and before this script is run. If this previous backup process is still running when this script attempts to run backup -q run again, then the second backup process will fail because the previous backup process is already running.

truncate.sh

This looks for and truncates excessively long archive directories that may have caused the backups of these archives to fail. See: truncate.sh.

rate.pl

This produces summaries of backup rates (seconds per user). See: rate.pl

This script is called twice:

1. With '-s' to append a rate summary to ~backups/logs/rate.s.log
2. With '-r' to append a progressive rate summary to ~backups/logs/rate.r.log

backupstat.new

This script attempts to gauge the success of the last NW Backups (run by backup above).

It does this by summarizing the dailysummary log file to which run_rsnapshot append a summary of each user's rsync backup summary. See: backupstat.new.

Note: It used to call backupstat, which would run a find over all the latest archives and count the number of archives with new/changed files in them.

Logs

~backups/logs/runbackups.log

Records the last time runbackups.sh was run, and which and when the various scripts were run.

backup

This is the main script that contains most of the bespoke commands used by the backup system.

Location

~backups/bin/backup

Called by

runbackups.sh

Many of the commands within backup can be run individually by passing them as arguments to backup, along with their desired options.

The backup -h option produces help documentation describing backup and its constituent commands. This documentation is duplicated below, with additional details where this might be helpful.

Manual Page

Usage

backup [-h] [-l log] [-n] [-q] command [commandargs]

Function

This script deals with CSE rsnapshot backups made for any CSE user who owns a home, or other directory, that is hosted on a CSE file server.

This script should usually be run on the CSE backup server: nw-syd-backup1.

Options

-h [ command | topic ]

Print help for the specific command or topic passed. If no command or topic is passed, print this general help and exit.

Topics and commands are summarised below.

-l logfile

Log messages to logfile (Default: ~backups/log/)

-n

Do not make any changes - just report what would be done.

-q

Do not reproduce messages on STDERR.

command [command_args]

Run the backup command with its optional command arguments.

• Backup commands are summarised below.
• Full descriptions of backup commands and their args may be produced by running:backups -h command. They are also included in what follows, under their own subsections.

Topics

general

Usage: backups [-h] [-l log] [-n] [-q] command [commandargs]

locations

Locations used by the CSE Backups System

overview

Overview of CSE Backup System

Commands

diff

User related diff.

fixprimary

Fix primary home archive names if necessary.

mkspec

Prepare backup archive directories and specfiles for users.

movearch

Copy backup archives to another backup filesystem.

movesource

Change the recorded source directory for a user's backup archive.

purge

Shifts and/or deletes archives of expired users.

redo_sync

Redo rsnapshot sync for archives reporting rsync errors.

rename

Renames archive(s) of a CSE user whose username has changed.

report

Summarise logs to report on the last backups.

resurrect

Resurrect a user's inactive backup archive from deleted archives.

run

Create and/or update CSE New World user backup archives.

run

This is the main backup command, which actually runs the rsnapshot backups for each CSE user. It is intended to be run once a day, and will usually take several hours to complete.

It records its process id in ~backups/var/runid, and checks that no other run process is already running before it starts. If another run process is already running, it exits with an error.

Usage

backup [-genopts] run [-m MAXSPEC] [-N[123]] [-n] [-p MAXPROC] [-s SCRIPT] [-u user]

Function

Create and/or update CSE New World user backup archives.

Details

1. Create lists of all CSE (user/directories) in ~backups/var/.

   • Classify these directory sources
   • Fix primary homes if necessary (See: fixprimary command below)

2. Create specfiles:

   • Ensure a backup archive exists for each (user/dir).

   (Archives are stored in $BU_USERS/)

   • Create rsnapshot specfiles for each (user/dir).

   (specfiles are stored in /var/tmp/backups/)

3. Use xargs to run SCRIPT on each specfile:

   • xargs runs MAXPROC simultaneous invocations of SCRIPT over batches of MAXSPEC specfiles.

   (See '-m', '-p' and '-s' options below)

4. Check for sync errors in log: ~backups/logs/rsnapshot.log

   • If errors are present, repeatedly run redo_sync until no further errors are recorded, or we have rerun redo_sync a maximum of 3 times.

Options

-D

Run an xargs process for each physical disk storing archives.

This attempts to evenly distribute simultaneous disk activity across all physical disks.

Default: Only run one xargs process, and ignore details of physical storage disks.

-m MAXSPEC

Each xargs process runs MAXPROC simultaneous invocations of SCRIPT over batches of MAXSPEC specfiles. After each batch of MAXSPEC archives has been processed, MAXPROC is reevaluated, and xargs used to invoke SCRIPT over the next MAXSPEC archives, using the reevaluated value of MAXPROC. (See '-p' option)

Default: xargs calls SCRIPT aver all available archives, with the value of MAXPROC being the one appropriate at initial invocation.

-N1

Do NOT (re)create user dir lists. (ie: Do not run Function step 1)

-N2

Do NOT (re)create rsnapshot dirs or specfiles. (ie: Do not run Function steps 2.1, 2.2)

-N3

Do NOT use xargs to call SCRIPT at all. (ie: Do not run Function step 3).

-n

Equivalent to -N3 or '-m 0'.

-p MAXPROC

Each xargs process runs MAXPROC simultaneous instances of SCRIPT.

The default value varies between 1 and 20, depending on:

1. The time of day
2. The number of physical disks, and
3. Whether '-D' option applies.

-s SCRIPT

xargs consecutively invokes this SCRIPT (in step 3), passing SCRIPT another specfile each time.

Default: ~backups/bin/run_rsnapshot:

1. Runs rsnapshot (possibly various times) passing specfile and various options.
2. Removes the specfile after rsnapshot has run.

Note: SCRIPT may be any program. It need not run rsnapshot, nor must it do anything with the specfile passed (although both of these things would be silly).

-u user

Create the specfiles for just this user (unless '-N2'),

Run SCRIPT passing just user's specfiles (unless '-n')

(Default: Create and run specfiles of all CSE users).

Files

• ~backups/var/runid: Process Id of current run process.
• ~backups/logs/backuplog:Trace of the run activity for each day of this month. Includes each overall run start and end time, the number of archives copied over NFS or SSH, the number of deleted archives shifted over or purged each day, and other miscellaneous details. This logfile is rotated monthly over seven months.
• ~backups/logs/rsnapshot.log:Output from all the rsnapshot processes run today, Includes when each archive started and finished, the details of the moves, copies, and rsyncs run. This logfile is rotated daily over seven days.

mkspec

Usage

backup [-genopts] mkspec [-D] [-nfs] [file]

Function

Prepare backup archive directories and specfiles for users specified in file.

• This is the function called by the run command to perform its Function steps 2.1 and 2.2.
• Read the input file of format: directory user host.

For each (user/dir) for which host is defined:

1. Find or create an rsnapshot archive for (user/dir) in ~backups/users/$LASTTWO/$user[.n]/.
2. Create individualised rsnapshot specfiles for each user/dir. (By default in /var/tmp/backups/)

If host is not defined, but an rsnapshot archive has been found for this (user/dir), then move the archive to ~backups/users/$LASTTWO/.deleted/$user[.n].

If host is not defined, but no rsnapshot archive is found, then silently ignore the (user/dir).

Options

-D

Create the specfiles in /var/tmp/backups/disk.N/ depending on the actual disk partition on which the archive is stored. This allows the run command to be run with the -D option.

-nfs

The rsnapshot specfile that is created, will specify that the directory should be accessed over NFS, rather than via an SSH connection to host (which is what is specified by default).

file

This is the input file of format: directory user [host].

If host is present then the directory exists on host.

(Default file: ~backups/var/ssh_dirs)

report

Usage

~backups/bin/backup [-genopts] report [-d date]

Function

Summarise logs to report on the last backups.

Options

-d date

Poduce a report on all backups run on this date.

Date is specified as yyyy.mm.dd

NOT YET IMPLIMENTED

diff

Usage

~backups/bin/backup [-genopts] diff user

Function

User related diff.

*NOT YET IMPLIMENTED*

movearch

Usage

~backups/bin/backup movearch [archname tofs | -f file]

Function

The backup archive archname is copied from its current archive filesystem to another.

The backup archive: ~backups/users/$LASTTWO/$archname

which is a soft link to its actual location at: ~backups/disks/N/$LASTTWO/$archname

is copied to a new location at: ~backups/disks/M/$LASTTWO/$archname.

If the copy is successfull, the soft link is adjusted to the new location.

Options

archname

The name of the specific archive to be moved

This is usually of the form: 'username[.[0-9]]'

tofs

Full pathname of the destination filesystem (eg: /export/nw-syd-backup1/1/backups/disks/3)

-f file

Read file of format: archname tofs, and copy and move each archive to tofs.

If file is -, then the script will read from STDIN.

Default: No options or args => Read from STDIN. (ie: -f - )

Note

This command does not change the recorded source of the archive (ie: the location of the user's actual home directory).

If you have moved the source of the archive from one home directory server/filesystem to another, then you should use the movesource command instead.

movesource

Usage

~backups/bin/backup movesource user old_source_pathname new_source_pathname

Function

Change the source_pathname recorded for a user's backup archive.

Details

The full source_pathname of each backup archive (ie: the full pathname of the user's actual home or other directory) is recorded in two places:

1. The file: ~backups/users/$LASTTWO/$username[.n]/.bu_source

   This file is written and checked by backup.

2. In each archive's retention level pathname:

   ~backups/users/$LASTTWO/$username[.n]/daily.N/$username/$source_pathname/

   The actual directory pathname matching source_pathname is written and checked by rsnapshot.

Use

Run this command when a user's home (or other directory) has had its pathname changed in any way, either because:

1. Some directories in the pathname were renamed (but the contents were not physically relocated), or
2. The directory and its contents were relocated/copied from one disk server's file system to another, resulting in a pathname change.

Example

~backups/bin/backup movesource zain /import/kamen/1/zain /import/glass/A/zain

Note

• This command only changes the archive source records in Details (1) and (2) above.

The actual relocation/renaming necessitating the movesource are expected to be done separately elsewhere.

• If $username has also changed, and not just the source_pathname, then you should also use the rename command.

redo_sync

Usage

backups redo_sync

Function

Redo rsnapshot sync for archives reporting rsync errors.

Details

Any archive that has returned an rsync error (possibly as a result of an intermittent network dropout) as recorded in the log:

/import/nw-syd-backup1/1/backups/logs/rsnapshot.log

will have its rsnapshot sync command rerun.

A checkpoint is written to the log marking the point at which this command was run. This allows previous errors addressed by this command to be ignored, and any subsequent errors (should they occur) to be considered anew.

Exit Value

0

No rsync errors to redo, or the redo created no subsequent errors.

1

Rsync errors existed, and the redo created at least one subsequent error.

rename

Usage

backup rename oldusername newusername

Function

Renames archive(s) of a CSE user whose username has changed.

Details

In general, backup archives belonging to username will include the following access pathnames:

nw-syd-backup1:/export/nw-syd-backup1/1/backups/users/$LASTTWO/$username[.n]/daily.N/$username/$source_pathname/

This command changes all occurences of $username in such access pathnames from oldusername to newusername, also changing $LASTTWO where necessary.

Note

This command will not change any further occurences of $username that may occur within $source_pathname. If $source_pathname has changed in any way, then you should also use the movesource command.

resurrect

Usage

backup resurrect user [sourcedir]

Function

Resurrect a user's inactive backup archive from deleted archives.

If sourcedir is passed, only resurrect the user's inactive backup archive coming from sourcedir, otherwise resurrect all inactive backup archives belonging to user.

Details

1. Active CSE users have backup archives kept in: ~backups/users/$LASTTWO/username[.n]/
2. Expired CSE users have their backup archives made inactive and moved to: ~backups/users/$LASTTWO/.deleted/username[.n]/

If an expired CSE user has their account reactivated, then this command reactivates their backup archive by moving it from (2) to (1).

Note

• This command only resurrects the backup archive (if it exists), so that the user's home directory(s) may be backed up once more.

It does not restore the user's original home directory from the backup archive. Run restore to do this.

• The source of the resurrected backup archives are assumed to stay unchanged.

If the user's original directory(s) is restored into different file systems/sourcedirs, in addition to resurrecting the user's original archive, you may need to use movesource and/or rename.

See Also

purge - for what happens to deleted archives.

movesource - for moving archive sources.

rename - for changing user names.

restore - for restoring user directory from archives.

purge

Usage

backup [-genopts] purge [-u user] [-m MAX] [-n]

Function

Shifts and/or deletes expired backup archives.

Details

When CSE users are expired and/or their home or other directory has been deleted, the run command moves the associated archives aside

from: ~backups/users/$LASTTWO/user[.n]/

to: ~backups/users/$LASTTWO/.deleted/user[.n]/.

Thereafter, each time this command is run, this command:

1. Shifts up the retention directories of all expired archives;
2. Removes any expired archives that have no retention directories left.

Options

-u user

Just shift and/or purge rsnapshot archives belonging to user.

(Default: Shift and/or purge all user archives in .deleted/ directories)

-m MAX

Only shift and/or purge MAX users.

-n

Do not actually shift or remove any rsnapshot archives. Just list archives that are affected on STDOUT.

-v

verbose listing - include disk usage

fixprimary

Usage

backup fixprimary

Function

Fix primary home archive names if necessary.

For every CSE user with:

• A CSE home and/or other directory, and
• At least one CSE backup archive of these home/directories,

Ensure that the user's primary home directory is stored in their primary backup archive (ie: their primary home directory is not stored in a backup archive with a .N suffix).

This command uses renamearch to rename archives if necessary.

run_rsnapshot

This script runs rsnapshot on the individual backup archive whose rsnapshot config file is passed to it. This script is usually run by Xargs(1) which in turn is run by backups run. Xargs(1) is passed a list of rsnapshot config files, which it passes to separate instances of this script, passing it a separate rsnapshot specfile each time.

This script, in turn, calls rsnapshot at least once (but possibly a few times in succession), passing the specfile each time to rsnapshot via the -c specfile option, before appending an appropriate snapshot type as the last command argument to rsnapshot.

Location

~backups/bin/run_rsnapshot

Called by

backup run

Rationale

Usually, rsnapshot is used to make snapshot backups of many users at once (possibly collecting from many different directory sources, and distributing to many different archive repositories). There are also usually several staggered cron entries on the one backup machine running different rsnapshot levels of each of these user/archive collections.

However, here at CSE, we wanted a separate rsnapshot archive for each user and their source directory, so that each rsnapshot archive could be run independently of all other rsnapshot archives, without the possibility of jeopardising these other rsnapshot archives should any individual archive fail to run properly. Of course, if we maintained one cron entry for each separate rsnapshot archive's run level, this would usually mean an unwieldy and impracticable number of cron entries to have to manage.

This script allows us to rsnapshot a large number of individual archives while avoiding having to have a separate cron entry per archive. It does this by effectively running all the cron tests in the script in succession on each rsnapshot archive that is passed, calling the correct successive rsnapshot level in the correct sequence when appropriate. This also has the advantage that each snapshot level is run after the previous level has completed properly (which does not necessarily happen if different levels are run by different cron entries).

Manual Entry

Usage

run_rsnapshot [-d] [-D date] configfile

Function

Run an appropriate series of rsnapshot calls:

rsnapshot -c configfile level

where level: (daily, weekly, monthly, yearly, sync)

Actions

1. Add user's rsnapshot logfile (recording files transferred) to user's rsnapshot configfile. Reset user's logfile.
2. Consecutively run a series of calls to different rsnapshot levels depending on:
   1. The current date;
   2. The options passed.
3. Append summary of user's logfile to the global daily summary log: ~backups/logs/dailysummary.
4. Remove user's configfile when finished.

Options

-d

configfile relates to a deleted user archive.

Deleted user archives:

• Are usually stored in: ~backups/users/$lasttwo/.deleted/$user.
• Are archives for users whose accounts have expired, and/or whose archived directory has either been deleted or moved aside into the holding directory: /home/file/system/.expired/.
• Should continue to age (ie: they should continue to move along from daily, to weekly, to monthly, to yearly)
• Are no longer synced from their original source directory. As such this script never performs the following actions for deleted archives:

1. Reset the user's logfile,
2. Run a rsnapshot sync level;
3. Append to the global summary.

-D date

Set today to date (eg: -D 2024/01/20).

This argument is expected to be used for testing and/or for specifically calling different levels (which are activated at specific dates).

configfile

The rsnapshot specfile that specifies the archive to be snapshoted - usually a single archive of a specific (usually home) directory belonging to one CSE user.

truncate.sh

This script truncates excessively long directories from archives that can cause rsnapshot(1) to fail.

Location

~backups/bin/truncate.sh

Called by

runbackup.sh

Manual Page

Usage

truncate.sh [-n] [-f] [-d depth] [-l outfile] [infile]

Function

This script truncates excessively long directories from archives that can cause rsnapshot(1) to fail.

Details

rsnapshot, the basis of CSE's New World Backup system, relies on the command cp to copy/link:

From the user's last backup directory archive (renamed daily.1/),

To the user's new backup directory archive (called daily.0/).

However, if daily.1/ contains any directories that are too deeply nested, then the recursive copy/link fails, and the rsnapshot of that archive fails after it records an error message in its logfile.

This script is intended to be run after rsnapshot is run. It:

1. Reads rsnapshot logfile infile, and looks for error messages that may indicate problems copying deeply nested directories.
2. Adds these potentially problematic archives to the list of previous archives that needed truncation: ~backups/logs/truncate.users.
3. Searches these archives for directories nested deeper than depth levels.
4. Truncates these directories within their last archive (daily.0) at depth.
5. Logs changes in ~backups/logs/truncate.log
6. Rewrites the archives needing truncation to ~backups/logs/truncate.users.

Note

• This means that no archive will have deeply nested directories, except for those deeply nested directories that have not yet been rsnapshot'ed a second time (requiring the copy and thereby creating the initial error message).
• This script is only necessary if cp keeps this depth related limitation.

Options

-n

Do not make changes or log anything to outfile. Just report what changes would have been made.

(Default: Do not make changes if truncate.users is newer that $outfile).

-f

Force changes to be made and logged to outfile.

Beware: truncate.sh should only be forced to make changes if backups has been run beforehand.

(Default: Make changes if and only if truncate.users if older than $outfile)

-d depth

The minimum depth of the pathnames to be truncated. Default: 200

-l outfile

Output log file. Must be an absolute pathname.

Default: ~backups/logs/truncate.users

-q

Quiet; If this script is making changes (ie: no -n) then touch outfile rather than append timestamp to outfile.

infile

Input file. May be a relative pathname. Default: ~backups/logs/rsnapshot.log

rate.pl

Location

~backups/bin/rate.pl

Called by

runbackup.sh

Usage

rate.pl [-s] [-l [-p pNO] | -r [-n MAX]] [file]

Function

Produce backup runtime stats (seconds per user) from either:

1. backup logs (default or -l), or
2. backup records (-r).

By default, this produces stats for each day from this month, including when the backup finished, how long it ran, the total number of archives processed, and the average number of seconds per user/archive.

Options

-l

Produce stats from backup logs.

This log contains records from this month's backup runs, and the default stats include (secs/user) for each day from this month.

This is the default, and excludes -r.

(Default file: ~backups/logs/backuplog)

-n MAX

Produce a rate calculation every MAX users.

Applies only to stats from rsnapshot logs.

(Default: 1000)

-p PNO

Only produce stats for process PNO.

Applies only to stats from backup logs, which include process id in logs.

-r

Produce stats from run_rsnapshot logs.

This log contains the single day's records for the last run of backups, and the default stats includes (secs/user) stats for every 1000 users.

This excludes -l;

(Default file: ~backups/logs/bu_record)

-s

Summary; Print the last (secs/user) stat only.

[file]

The file of records to be read

'-' reads from STDIN.

backupstat.new

Location

~backups/bin/backupstat.new

Called by

runbackup.sh

Usage

backupstat.new {logfile};

Function

Summarize the log of the most recent user archive changes in: /export/nw-syd-backup1/1/backups/logs/dailysummary

dailysummary lists all the archives last backed up, with the number of new or modified files written to each archive by rsync.

This script summarises dailysummary by printing the total number of archives last backed up, and the number of these archives that had at least one new or modified file written to it.

To be considered successfull, the total number of archives should very similar from day to day (+/- the new additions and old removals), while the second number should be between a quarter to a third of the total (depending on the day of the week, and how many assignments are expected around then). This is currently the best gauge of the success (or failure) of the last New World Backups.

Details

The script:

run_rsnapshot

runs rsnapshot for each user, configuring it such that rsync logs the changes to the user's archive in the log file:

$lasttwo/user/log

Once the rsnapshot (and rsync) of the user has finished, run_rsnapshot appends a summary of the user's rsync log to:

$BU/logs/dailysummary

Finally, runbackup.sh runs this script which summarises dailysummary. runbackup.sh appends this summary to:

$BU/logs/backupstat.log

backupstat

Location

~backups/bin/backupstat

Called by

Nothing. Used to be called by: runbackup.sh

Usage

backupstat [-N] ( [-f] [-n] [-q] {logfile} | [-u user] )

Function

This script was first attempt to gauge the success of the last New World Backup run. Its function has been replaced by backupstat.new

The script reports on the number of CSE user archives in:

~backups/users/

that have at least one new (day old/changed) file in their archive:

daily.0/

Note:

• Given the conditions:

1. Backups have failed, and
2. N = Number of archives containing any new/changed files.
3. (1) implies (2) N=0, but
4. (2) N=0 does not imply (1)

• We want to determine (1) but can only measure (2).
• Despite (4), (2) is still a good measurement to make.

Details

This script:

1. For each user, looks for files that are either:
   1. Less than one day old (Default).
   2. Have only one link (-l) - ie: new/changed since last backup.

   Adds user's result to logfile: ~backups/logs/dayold

2. Summarizes the logfile.

By default, step (1) is only performed and the log file recreated if the logfile was not already modified today.

(See also options below to change this behaviour).

Options

-l

Find files with only one link (ie: the file is new or changed since the previous backup). This is a faster test than looking for files less than a day old (the default), but if backups had failed to run altogether, then daily.0/ would not represent today's archive, but the previous run's archive, and this test would duplicate the previous run's results (producing a false pass).

Default: Find files less than one day old. This is a slower test, but not susceptible to the link test's false pass result.

-u user

List all matching files in user's archive.

Ignore all other options except -l and do not read or write logfiles.

-c

Count all files found (slow).

Default: Quit after finding the first matching file - ie: produce 1 or 0 (fast).

-f

Force logfile to be recreated even if it was modified today.

-n

Do not recreate log file, even if it was not modified today.

This just produces the summary report from the existing log file.

This option overrides '-f'.

-q

Quiet - just produce summary.

logfile

Produce a report from logfile(s). Implies '-n'.

Note: logfile(s) may be compressed (.gz).

Default: ~backups/logs/dayold

bustatus.sh

Location

~backups/bin/bustatus.sh

Called by

Nothing. Intended to be called by the system monitoring script(s).

Usage

bustatus.sh [-q]

Function

Report on the current status of New World Backup System.

Any reported failure should be investigated further and confirmed.

Options

-q

Quiet - Only return exit code. Do not print log summaries.

Exit codes

   0	Success
   1	Failure
   2	Still running