New World Backup System: Difference between revisions

From techdocs
Jump to navigation Jump to search
Line 176: Line 176:
::*With    '-D': 1;
::*With    '-D': 1;
::*Without '-D': 6
::*Without '-D': 6
:;-s SCRIPT:''xargs'' consecutively invokes this ''SCRIPT'' (in step 3), passing ''SCRIPT'' the specfile for a different user/archive each time.
:;-s SCRIPT:''xargs'' repeatedly invokes this ''SCRIPT'' (in step 3), passing it the specfile for a different user/archive each time.
::Default ''SCRIPT'': <code>~backups/bin/run_rsnapshot</code> (See [[#run_rsnapshot|run_rsnapshot]])
::Default ''SCRIPT'': <code>~backups/bin/run_rsnapshot</code> (See [[#run_rsnapshot|run_rsnapshot]])
::Note: ''SCRIPT'' may be any program. It need not run rsnapshot, nor must it do anything with the specfile passed (although both of these things would be silly).
::Note: ''SCRIPT'' may be any program. It need not run rsnapshot, nor must it do anything with the specfile passed (although both of these things would be silly).

Revision as of 13:25, 10 April 2024

Overview

Overall Brief
The following was the original overall brief for the CSE New World (NW) Backup System :
  1. The Backup system should make a separate archive for every CSE directory that "belongs" to to every CSE user.
  2. Each archive should be easily accessible by the user whose directory is being archived.
  3. Each separate archive should be independent of each other.
  4. Access to a user's archive directories and files should be protected by access permissions at least as strong as those used to protect the user's original source directories and files.
  5. New CSE users and/or directories should be able to be added to the backup system, and existing users and/or directories removed from it, without affecting any of the other existing user's archives.
  6. A series of daily snapshots should be made of each user's directory, which may subsequently by summarised and stored in a series of weekly, monthly, and yearly archives.
Current Configuration
In the Current configuration, there are progressively:
  • 7 daily archives
  • 4 weekly archives
  • 12 monthly archives
  • 1 yearly archive
Technology
The NW Backup System is built around rsnapshot(1) as its underlying technology
  • rsnapshot is a remote filesystem snapshot utility that uses rsync(1);
  • rsync is a fast and versatile, remote (and local) file and directory copying tool that
  • Handles local system snapshots directly, and
  • Handles remote system snapshots using ssh(1).
  • Typically, rsnapshot is set up to store snapshot copies of source directories in an archive filesystem.
Periodically (typically each day) rsnapshot uses rsync to make a new snapshot of each of its source directories:
  1. Copying to the new snapshot archive only those files that are new, or which have changed, since the last copy was made;
  2. Retaining links in the new snapshot archive to those files in the previous archive that remain unchanged.
  • The CSE backup system is run on the New World AWS backup machine:
nw-syd-backup1.
  • The scripts, logs, and archives associated with NW backups are generally stored in the directory on the backup machine:
nw-syd-backup1:/export/nw-syd-backup1/1/backups/ (aka: ~backups)

Operation

Archive Directories
Each CSE directory on a CSE fileserver belonging to a CSE user is backed up and stored in the user's separate rsnapshot Archive Directory:
~backups/users/$LASTTWO/$user[.n]/
where:
  • $LASTTWO is the last two characters of the user's CSE username,
  • $user is the name of the CSE user whose directory is stored in the archive.
  • [.n] is a unique numeric suffix assigned to each of the user's different CSE directories (if the user has more than one).
Identity file
The file: $user[.n]/.bu_source in the archive directory is used to identify the user's CSE directory that is stored in the archive directory.
Retention Directories
Copies (or snapshots) are made of the user's directory and kept in various Retention Directories within each user's Archive Directory. These retention directories are named:
  1. daily.0/ .. daily.6/
  2. weekly.0/ .. weekly.3/
  3. monthly.0/ .. monthly.11/
  4. yearly.0/
  • In general, each retention directory indicates how long ago the snapshot archive that it contains was made. For example daily.3/ contains a copy of the user's directory that was made 4 days ago, whereas weekly.2/ will contain a copy of the user's directory made 3 weeks ago.
  • The daily.N/ directories are shifted up to daily.N+1/ each day that they age, while daily.6 is either shifted up into weekly.0 (once a week), or else it is deleted.
  • Similar progressive shifts and/or deletions occur for each other set of retention directories.
Access
The same access (ie: ownership and permissions) is assigned to each file and directory in the user's backup archive as were assigned to each file and directory in their original CSE (home) directory.
Note however:
  • The filesystems storing the CSE archives are mounted read only outside of the backup server,
  • This means that no user can change the contents or access permissions of any file or directory in their archive, even if they seem to have permissions to do so.
  • This means that it may be possible for users to not be allowed access to files or directories in their archive, if they were originally copied with the wrong access permissions.
In this case, the users will need to get the help of the system staff (root) to access these files/directories.
Account Expiry
Once a user's CSE account has expired, and/or the user's CSE directory has been removed from its host server, the user's archive of their directory will also be moved aside
from: ~backups/users/$LASTTWO/$user[.n]/
to: ~backups/users/$LASTTWO/.deleted/$user[.n]/.
However:
  • The retention directories will continue to be shifted up as appropriate until they have all been removed.
  • If the user's CSE directory is restored to the CSE host sever before all retention dirs have been removed, then the most recent archive directory is renamed daily.0/, and the remaining directories retained as is.

Locations

CSE's backup server
nw-syd-backup1@cse.unsw.edu.au
~backups
nw-syd-backup1:/export/nw-syd-backup1/1/backup/
~backups/users/$LASTTWO/$user[.n]/
This is the location of $user's rsnapshot archives, such that:
  • $LASTTWO is the last two characters of the user's CSE username,
  • $user is the name of the CSE user whose directory is stored in the archive.
  • If the user has been allocated more than one CSE directory, then [.n] is a unique numeric suffix assigned to all but one of the user's CSE directories.
The archive without the numeric suffix should always be the archive that stores the user's main home directory.
  • $user[.n] is actually a soft link to the actual storage location of the backup archive: backups/disks/$mountpoint/$LASTTWO/$user[.n]
~backups/users/$LASTTWO/$user[.n]/.bu_source
This file identifies the source of the actual user's directory that is stored in this rsnapshot archive. It will mpst likely be of the form: /import/bach/1/username
~backups/users/$LASTTWO/.deleted/$user[.n]/
If $user's source directory, as identified by the file: $user[.n]/.bu_source, no longer exists on the source filesystem, then the user's archive is moved into this .deleted/ directory.
~backups/disks/$mountpoint/$LASTTWO/$user[.n]/
This is the actual storage location of the backup archive for $user[.n].
There may be many different $mountpoint each of which represents a separate (possibly virtual) disk storage filesystem on which user's archives may be stored. This provides a way of managing disk space allocation.
The directory: ~backups/users/$LASTTWO/$user[.n]/ is actually a symlink to the actual directory in ~backups/disks/$mountpoint/$LASTTWO/$user[.n] that has been allocated to this user's backup archive.
~backups/var/
The location of various files containing (user/dir) data.
These files group (user/dir) data into different classifications which are used to determine which (user/dir)s are to be snapshotted, and how their archives are to be managed.
~backups/lib/
The location of most of the configuration files used by the NW backup system.
~backups/bin/
The location of most of the scripts used by the backup system, including the main backup script itself (described in the next section).

Scripts

runbackup.sh

This is the main script tying all the other scripts together that relate to backups.

Location
~backups/bin/runbackups.sh
Called by
nw-syd-backup1:/etc/cron.d/rsnapshot
Calls
logrotate(1)
This specifically rotates those logs in ~backups/logs/ that have anything to do with new backups.
logrotate is passed the specfile: ~backups/lib/logrotate.conf
backup -q run -p 5
This is the main backup command that updates the backup archives. See: backup and run.
truncate.sh
This looks for and truncates excessively long archive directories that may have caused the backups of these archives to fail. See: truncate.sh.
rate.pl
This produces summaries of backup rates (seconds per user). See: rate.pl
This script is called twice:
  1. With '-s' to append a rate summary to ~backups/logs/rate.s.log
  2. With '-r' to append a progressive rate summary to ~backups/logs/rate.r.log
backupstat
This script attempts to gauge the success of the last NW Backups (run by backup above).
It does this by counting the number of archives with new/changed files in them. See: backupstat.
This script is called twice:
  1. With '-f' to force the script to find and count the new/changed files in each archive.
  2. With '-n -q' to append the summary of the last find to ~backups/logs/rate.r.log
Logs
~backups/logs/runbackups.log
Records the last time runbackups.sh was run, and which and when the various scripts were run.

backup

This is the main script that contains most of the bespoke commands used by the backup system.

Location
~backups/bin/backup
Called by
runbackups.sh

Many of the commands within backup can be run individually by passing them as arguments to backup, along with their desired options.

The backup -h option produces help documentation describing backup and its constituent commands. This documentation is duplicated below, with additional details where this might be helpful.

Manual Page

Usage
backup [-h] [-l log] [-n] [-q] command [commandargs]
Function
This script deals with CSE rsnapshot backups made for any CSE user who owns a home, or other directory, that is hosted on a CSE file server.
This script should usually be run on the CSE backup server: nw-syd-backup1.
Options
-h [ command | topic ]
Print help for the specific command or topic passed. If no command or topic is passed, print this general help and exit.
Topics and commands are summarised below.
-l logfile
Log messages to logfile (Default: ~backups/log/)
-n
Do not make any changes - just report what would be done.
-q
Do not reproduce messages on STDERR.
command [command_args]
Run the backup command with its optional command arguments.
  • Backup commands are summarised below.
  • Full descriptions of backup commands and their args may be produced by running:backups -h command. They are also included in what follows, under their own subsections.
Topics
general
Usage: backups [-h] [-l log] [-n] [-q] command [commandargs]
locations
Locations used by the CSE Backups System
overview
Overview of CSE Backup System
Commands
diff
User related diff.
fixprimary
Fix primary home archive names if necessary.
mkspec
Prepare backup archive directories and specfiles for users.
movearch
Copy backup archives to another backup filesystem.
[[#movesource|movesource]
Change the recorded source directory for a user's backup archive.
purge
Shifts and/or deletes archives of expired users.
rename
Renames archive(s) of a CSE user whose username has changed.
report
Summarise logs to report on the last backups.
resurrect
Resurrect a user's inactive backup archive from deleted archives.
run
Create and/or update CSE New World user backup archives.

run

This is the main backup command, which actually runs the rsnapshot backups for each CSE user. It is intended to be run once a day, and will usually take several hours to complete.

It records its process id in ~backups/var/runid, and checks that no other run process is already running before it starts. If another run process is already running, it exits with an error.

Usage

backup [-genopts] run [-m MAXSPEC] [-N[123]] [-n] [-p MAXPROC] [-s SCRIPT] [-u user]

Function
Create and/or update CSE New World user backup archives.
  1. Create lists of all CSE (user/directories) in ~backups/var/.
    • Classify these directory sources
    • Fix primary homes if necessary (See: fixprimary command below)
  2. Create specfiles
    1. Ensure a backup archive exists for each (user/dir).
      (archives in $BU_USERS/)
    2. Create rsnapshot specfiles for each (user/dir).
      (specfiles in /var/tmp/backups/)
  3. Use xargs to run consecutive invokations of SCRIPT, passing each process a different specfile for a different user/archive.
    (See '-s SCRIPT' option below)
Options
-D
Run an xargs process for each physical disk storing archives.
This attempts to evenly distribute simultaneous disk activity across all physical disks.
Default: Only run one xargs process, and ignore details of physical storage disks.
-m MAXSPEC
Each xargs process will call SCRIPT no more than MAXSPEC times.
Default: Call SCRIPT until all specfiles are processed.
-N1
Do NOT (re)create user dir lists. (ie: Do not run Function step 1)
-N2
Do NOT (re)create rsnapshot dirs or specfiles. (ie: Do not run Function steps 2.1, 2.2)
-N3
Do NOT use xargs to call SCRIPT at all. (ie: Do not run Function step 3).
-n
Equivalent to -N3 or '-m 0'.
-p MAXPROC
Each xargs runs MAXPROC simultaneous process sequences of SCRIPT.
Default:
  • With '-D': 1;
  • Without '-D': 6
-s SCRIPT
xargs repeatedly invokes this SCRIPT (in step 3), passing it the specfile for a different user/archive each time.
Default SCRIPT: ~backups/bin/run_rsnapshot (See run_rsnapshot)
Note: SCRIPT may be any program. It need not run rsnapshot, nor must it do anything with the specfile passed (although both of these things would be silly).
-u user
Create the specfiles for just this user (unless '-N2'),
Run SCRIPT passing just user's specfiles (unless '-n')
(Default: Create and run specfiles of all CSE users).

Files

  • ~backups/var/runid: Process Id of current run process.
  • ~backups/logs/backuplog:Trace of the run activity for each day of this month. Includes each overall run start and end time, the number of archives copied over NFS or SSH, the number of deleted archives shifted over or purged each day, and other miscellaneous details. This logfile is rotated monthly over seven months.
  • ~backups/logs/rsnapshot.log:Output from all the rsnapshot processes run today, Includes when each archive started and finished, the details of the moves, copies, and rsyncs run. This logfile is rotated daily over seven days.

mkspec

Usage
backup [-genopts] mkspec [-D] [-nfs] [file]
Function
Prepare backup archive directories and specfiles for users specified in file.
  • This is the function called by the run command to perform its Function steps 2.1 and 2.2.
  • Read the input file of format: directory user host.
For each (user/dir) for which host is defined:
  1. Find or create an rsnapshot archive for (user/dir) in ~backups/users/$LASTTWO/$user[.n]/.
  2. Create individualised rsnapshot specfiles for each user/dir. (By default in /var/tmp/backups/)
If host is not defined, but an rsnapshot archive has been found for this (user/dir), then move the archive to ~backups/users/$LASTTWO/.deleted/$user[.n].
If host is not defined, but no rsnapshot archive is found, then silently ignore the (user/dir).
Options
-D
Create the specfiles in /var/tmp/backups/disk.N/ depending on the actual disk partition on which the archive is stored. This allows the run command to be run with the -D option.
-nfs
The rsnapshot specfile that is created, will specify that the directory should be accessed over NFS, rather than via an SSH connection to host (which is what is specified by default).
file
This is the input file of format: directory user [host].
If host is present then the directory exists on host.
(Default file: ~backups/var/ssh_dirs)

report

Usage
~backups/bin/backup [-genopts] report [-d date]
Function
Summarise logs to report on the last backups.
Options
-d date
Poduce a report on all backups run on this date.
Date is specified as yyyy.mm.dd

NOT YET IMPLIMENTED

diff

Usage
~backups/bin/backup [-genopts] diff user
Function
User related diff.

*NOT YET IMPLIMENTED*

movearch

Usage
~backups/bin/backup movearch [archname tofs | -f file]
Function
The backup archive archname is copied from its current archive filesystem to another.
The backup archive: ~backups/users/$LASTTWO/$archname
which is a soft link to its actual location at: ~backups/disks/N/$LASTTWO/$archname
is copied to a new location at: ~backups/disks/M/$LASTTWO/$archname.
If the copy is successfull, the soft link is adjusted to the new location.
Options
archname
The name of the specific archive to be moved
This is usually of the form: 'username[.[0-9]]'
tofs
Full pathname of the destination filesystem (eg: /export/nw-syd-backup1/1/backups/disks/3)
-f file
Read file of format: archname tofs, and copy and move each archive to tofs.
If file is -, then the script will read from STDIN.
Default: No options or args => Read from STDIN. (ie: -f - )
Note
This command does not change the recorded source of the archive (ie: the location of the user's actual home directory).
If you have moved the source of the archive from one home directory server/filesystem to another, then you should use the movesource command instead.

movesource

Usage
~backups/bin/backup movesource user old_source_pathname new_source_pathname
Function
Change the source_pathname recorded for a user's backup archive.
Details
The full source_pathname of each backup archive (ie: the full pathname of the user's actual home or other directory) is recorded in two places:
  1. The file: ~backups/users/$LASTTWO/$username[.n]/.bu_source
    This file is written and checked by backup.
  2. In each archive's retention level pathname:
    ~backups/users/$LASTTWO/$username[.n]/daily.N/$username/$source_pathname/
    The actual directory pathname matching source_pathname is written and checked by rsnapshot.
Use
Run this command when a user's home (or other directory) has had its pathname changed in any way, either because:
  1. Some directories in the pathname were renamed (but the contents were not physically relocated), or
  2. The directory and its contents were relocated/copied from one disk server's file system to another, resulting in a pathname change.
Example
~backups/bin/backup movesource zain /import/kamen/1/zain /import/glass/A/zain
Note
  • This command only changes the archive source records in Details (1) and (2) above.
The actual relocation/renaming necessitating the movesource are expected to be done separately elsewhere.
  • If $username has also changed, and not just the source_pathname, then you should also use the rename command.

rename

Usage
backup rename oldusername newusername
Function
Renames archive(s) of a CSE user whose username has changed.
Details
In general, backup archives belonging to username will include the following access pathnames:
nw-syd-backup1:/export/nw-syd-backup1/1/backups/users/$LASTTWO/$username[.n]/daily.N/$username/$source_pathname/
This command changes all occurences of $username in such access pathnames from oldusername to newusername, also changing $LASTTWO where necessary.
Note
This command will not change any further occurences of $username that may occur within $source_pathname. If $source_pathname has changed in any way, then you should also use the movesource command.

resurrect

Usage
backup resurrect user [sourcedir]
Function
Resurrect a user's inactive backup archive from deleted archives.
If sourcedir is passed, only resurrect the user's inactive backup archive coming from sourcedir, otherwise resurrect all inactive backup archives belonging to user.
Details
  1. Active CSE users have backup archives kept in: ~backups/users/$LASTTWO/username[.n]/
  2. Expired CSE users have their backup archives made inactive and moved to: ~backups/users/$LASTTWO/.deleted/username[.n]/
If an expired CSE user has their account reactivated, then this command reactivates their backup archive by moving it from (2) to (1).
Note
  • This command only resurrects the backup archive (if it exists), so that the user's home directory(s) may be backed up once more.
It does not restore the user's original home directory from the backup archive. Run restore to do this.
  • The source of the resurrected backup archives are assumed to stay unchanged.
If the user's original directory(s) is restored into different file systems/sourcedirs, in addition to resurrecting the user's original archive, you may need to use movesource and/or rename.
See Also
purge - for what happens to deleted archives.
movesource - for moving archive sources.
rename - for changing user names.
restore - for restoring user directory from archives.

purge

Usage
backup [-genopts] purge [-u user] [-m MAX] [-n]
Function
Shifts and/or deletes expired backup archives.
Details
When CSE users are expired and/or their home or other directory has been deleted, the run command moves the associated archives aside
from: ~backups/users/$LASTTWO/user[.n]/
to: ~backups/users/$LASTTWO/.deleted/user[.n]/.
Thereafter, each time this command is run, this command:
  1. Shifts up the retention directories of all expired archives;
  2. Removes any expired archives that have no retention directories left.
Options
-u user
Just shift and/or purge rsnapshot archives belonging to user.
(Default: Shift and/or purge all user archives in .deleted/ directories)
-m MAX
Only shift and/or purge MAX users.
-n
Do not actually shift or remove any rsnapshot archives. Just list archives that are affected on STDOUT.
-v
verbose listing - include disk usage

fixprimary

Usage
backup fixprimary
Function
Fix primary home archive names if necessary.
For every CSE user with:
  • A CSE home and/or other directory, and
  • At least one CSE backup archive of these home/directories,
Ensure that the user's primary home directory is stored in their primary backup archive (ie: their primary home directory is not stored in a backup archive with a .N suffix).
This command uses renamearch to rename archives if necessary.

run_rsnapshot

backups run uses Xargs(1) to run many separate instances of this script, passing it a separate rsnapshot specfile each time.

This script, in turn, calls rsnapshot at least once (but possibly a few times in succession), passing the specfile each time to rsnapshot via the -c specfile option, before appending an appropriate snapshot type as the last command argument to rsnapshot.

Rationale

Usually, rsnapshot is used to make snapshot backups of many users at once (possibly collecting from many different directory sources, and distributing to many different archive repositories). There are also usually several staggered cron entries on the one backup machine running different rsnapshot levels of each of these user/archive collections.

However, here at CSE, we wanted a separate rsnapshot archive for each user and their source directory, so that each rsnapshot archive could be run independently of all other rsnapshot archives, without the possibility of jeopardising these other rsnapshot archives should any individual archive fail to run properly. Of course, if we maintained one cron entry for each separate rsnapshot archive's run level, this would usually mean an unwieldy and impracticable number of cron entries to have to manage.

This script allows us to run a large number of individual archives while avoiding having to have a separate cron entry per archive. It does this by effectively running the cron tests in the script for every individual rsnapshot archive that is passed, calling the correct successive rsnapshot level in the correct sequence as appropriate.

Manual Entry

Usage
run_rsnapshot [-d] [-D date] configfile
Function
Run an appropriate series of rsnapshot calls:
rsnapshot -c configfile level
where level: (daily, weekly, monthly, yearly, sync)
Actions
  1. Reset user's logfile of files transferred.
  2. Run a series of rsnapshot calls depending on:
    1. The current date;
    2. The options passed.
  3. Append summary of user's logfile to the global daily summary log: ~backups/logs/dailysummary.
  4. Remove configfile when finished.
Options
-d
configfile is a deleted user archive. For such deleted archives, never run a sync level, reset the user's logfile, or append to the global summary.
-D date
Set today to date (eg: -D 2024/01/20).
Expected to be used for testing and/or for specifically calling different levels.
configfile
The rsnapshot specfile that specifies a single archive to be snapshoted - usually that of a directory belonging to one CSE user.

truncate.sh

This script truncates excessively long directories from archives that can cause rsnapshot(1) to fail.

Location
~backups/bin/truncate.sh
Called by
runbackup.sh

Manual Page

Usage
truncate.sh [-n] [-f] [-d depth] [-l outfile] [infile]
Function
This script truncates excessively long directories from archives that can cause rsnapshot(1) to fail.
Details
rsnapshot, the basis of CSE's New World Backup system, relies on the command cp to copy/link:
From the user's last backup directory archive (renamed daily.1/),
To the user's new backup directory archive (called daily.0/).
However, if daily.1/ contains any directories that are too deeply nested, then the recursive copy/link fails, and the rsnapshot of that archive fails after it records an error message in its logfile.
This script is intended to be run after rsnapshot is run. It:
  1. Reads rsnapshot logfile infile, and looks for error messages that may indicate problems copying deeply nested directories.
  2. Adds these potentially problematic archives to the list of previous archives that needed truncation: ~backups/logs/truncate.users.
  3. Searches these archives for directories nested deeper than depth levels.
  4. Truncates these directories within their last archive (daily.0) at depth.
  5. Logs changes in ~backups/logs/truncate.log
  6. Rewrites the archives needing truncation to ~backups/logs/truncate.users.
Note
  • This means that no archive will have deeply nested directories, except for those deeply nested directories that have not yet been rsnapshot'ed a second time (requiring the copy and thereby creating the initial error message).
  • This script is only necessary if cp keeps this depth related limitation.
Options
-n
Do not make changes or log anything to outfile. Just report what changes would have been made.
(Default: Do not make changes if truncate.users is newer that $outfile).
-f
Force changes to be made and logged to outfile.
Beware: truncate.sh should only be forced to make changes if backups has been run beforehand.
(Default: Make changes if and only if truncate.users if older than $outfile)
-d depth
The minimum depth of the pathnames to be truncated. Default: 200
-l outfile
Output log file. Must be an absolute pathname.
Default: ~backups/logs/truncate.users
-q
Quiet; If this script is making changes (ie: no -n) then touch outfile rather than append timestamp to outfile.
infile
Input file. May be a relative pathname. Default: ~backups/logs/rsnapshot.log

rate.pl

Location
~backups/bin/rate.pl
Called by
runbackup.sh
Usage
rate.pl [-s] [-l [-p pNO] | -r [-n MAX]] [file]
Function
Produce backup runtime stats (seconds per user) from either:
  1. backup logs (default or -l), or
  2. backup records (-r).
By default, this produces stats for each day from this month, including when the backup finished, how long it ran, the total number of archives processed, and the average number of seconds per user/archive.
Options
-l
Produce stats from backup logs.
This log contains records from this month's backup runs, and the default stats include (secs/user) for each day from this month.
This is the default, and excludes -r.
(Default file: ~backups/logs/backuplog)
-n MAX
Produce a rate calculation every MAX users.
Applies only to stats from rsnapshot logs.
(Default: 1000)
-p PNO
Only produce stats for process PNO.
Applies only to stats from backup logs, which include process id in logs.
-r
Produce stats from run_rsnapshot logs.
This log contains the single day's records for the last run of backups, and the default stats includes (secs/user) stats for every 1000 users.
This excludes -l;
(Default file: ~backups/logs/bu_record)
-s
Summary; Print the last (secs/user) stat only.
[file]
The file of records to be read
'-' reads from STDIN.

backupstat

Location
~backups/bin/backupstat
Called by
runbackup.sh
Usage
backupstat [-N] ( [-f] [-n] [-q] {logfile} | [-u user] )
Function
Attempt to gauge the success of the last New World Backup run.
Report on the number of CSE user archives in:
~backups/users/
that have at least one new (day old/changed) file in their archive:
daily.0/
Note:
  • Given the conditions:
  1. Backups have failed, and
  2. N = Number of archives containing any new/changed files.
  3. (1) implies (2) N=0, but
  4. (2) N=0 does not imply (1)
  • We want to determine (1) but can only measure (2).
  • Despite (4), (2) is still a good measurement to make.
Details
This script:
  1. For each user, looks for files that are either:
    1. Less than one day old (Default).
    2. Have only one link (-l) - ie: new/changed since last backup.
    Adds user's result to logfile: ~backups/logs/dayold
  2. Summarizes the logfile.
By default, step (1) is only performed and the log file recreated if the logfile was not already modified today.
(See also options below to change this behaviour).
Options
-l
Find files with only one link (ie: the file is new or changed since the previous backup). This is a faster test than looking for files less than a day old (the default), but if backups had failed to run altogether, then daily.0/ would not represent today's archive, but the previous run's archive, and this test would duplicate the previous run's results (producing a false pass).
Default: Find files less than one day old. This is a slower test, but not susceptible to the link test's false pass result.
-u user
List all matching files in user's archive.
Ignore all other options except -l and do not read or write logfiles.
-c
Count all files found (slow).
Default: Quit after finding the first matching file - ie: produce 1 or 0 (fast).
-f
Force logfile to be recreated even if it was modified today.
-n
Do not recreate log file, even if it was not modified today.
This just produces the summary report from the existing log file.
This option overrides '-f'.
-q
Quiet - just produce summary.
logfile
Produce a report from logfile(s). Implies '-n'.
Note: logfile(s) may be compressed (.gz).
Default: ~backups/logs/dayold