CSE Account creation and maintenance scripts

From techdocs
Jump to navigation Jump to search

The UDB =

The CSE User DataBase (usually referred to as the UDB), is a PostGres database that stores the account details of all CSE users. It is usually accessed and modified using the acc command, although it can also be accessed and modified directly (with appropriate access permissions) using the postgres command pgsql.

This document describes the system scripts that are used to update the UDB from official student enrolment records, or from staff allocation and HR records.

/home/ss/accounts/bin/UDBupdate

Function
This is the main script that updates CSE student and staff accounts. It ties together and calls most of the other account related scripts.
  • Currently run nightly on the OW machine:synth by the maint system:
    synth:/usr/local/maint/nightly/S10_NSSupdate ->
    /home/ss/accounts/bin/NSSupdate ->
    UDBupdate
Note that the maint system is called at 2am on synth by /usr/local/maint/run nightly which is run by cron.
  • Its activity is logged in: /home/ss/logs/NSSupdate.$MONTH (eg: NSSupdate.Dec).
    The log records when each constituant script below is called, and captures their output.
  • UDBupdate, and most of the scripts called by it, may be run with:
    • The '-h' option, which produces specific and more detailed help information related to the script.
    • The '-n' option, which reports any changes to be made, but does not actually make any.
  • Most of the scripts run by UDBupdate may be run independently of UDBupdate. This is often useful when debugging problems.
UDBupdate should be unhooked from the maint system in the Old world.

What follows is a description of the scripts called by UDBupdate, in the order in which they are called.

Dealing with student records and accounts:

/home/ss/accounts/bin/tape.schedule

Function
List the SiMs files expected to be downloaded today.
  • This script needs to be updated whenever the parameters of the NSS downloads are changed (usually sometime before the start of each session).
  • It documents when (ie: day of week or month) that each download is expected to be run, and includes the details of the download parameters.
  • Contains details of how and where to log in to change download parameters.
    This is particularly useful.

/home/sturec/TAPES/bin/nss_sftp.pl

Function
perl script that uses sftp to copy NSS downloads to /home/sturec/TAPES/
  • I believe that this script needs to be run on synth because file.sims.unsw.edu.au did not used to accept ssh connections from any other CSE machine.
    This needs to be confirmed.
  • Currently, synth is a conformed machine, and this will eventually need to be changed.

/home/sturec/TAPES/bin/mv_nssreports.sh

Function
Give downloaded NSS data files standard names (in /home/sturec/TAPES) that are used/known elsewhere in CSE (by sms, teachadmin, other (ss) scripts).

Names are of the form:

YYTT_SCHOOL.TYPE
where:
  YY     year (last two digits of year);
  TT     term (T[0123] | H[1-6])
  SCHOOL Organisational unit 'owning' the records (eg: COMPSC,ENG,MULTI).
  TYPE   Download records type:
     enr     NSSR3502 enrolment records
     STDNT   NSSR5046 Student records
     SUBJ    NSSR5046subject records
     sched   NSSR2922 schedule records
     griff   Griffin records

mv_nssreports.sh uses:

  • /home/sturec/bin/extract.pl
    perl script which extracts data fields from standard downloads given field specifications (in /home/sturec/lib/field.*)
  • /home/ss/bin/unsw_dates.pl
    perl script generating dates for UNSW term start,end,expiry,etc.

/home/sturec/bin/mksturecdata

Function
Generate a single form of program or course enrolment data (generally one line per student course or program enrolment), from the different forms of download files obtained by nss_sftp.pl and renamed in mv_nssreports.sh.

These files are kept in:

/home/sturec/DATA/CCYYTT.(program|course)[.full]

Note: Many standard download files contain the same data in different formats. This script attempts to create single enrolment files with a uniform and consistent format from the many different download files.

/home/ss/accounts/bin/updatestu.pl

Function
Update the student records in the CSE UDB from enrolment records in /home/sturec/DATA/.

See the separate documentation below.

/home/ss/accounts/bin/updateold stu

Function
Update the CSE's old UDB from CSE's new UDB.
  • Make the same changes to the old UDB (using '/home/ss/accounts/bin/updatestu')
    that have already been made to the new UDB (with '/home/ss/accounts/bin/updatestu -N').
    1. Use old acc to update the old UDB using '/home/ss/accounts/bin/updatestu -n'
      This creates the file: 'acc.out'.
    2. Create a list of new users in 'acc.out' that don't yet exist in the old UDB.
    3. Extract the (uid, home, and name) of these new users who should already exist in the new UDB, and add these details to 'acc.out'.
    4. Run amended 'acc.out' through old acc to make changes in the old UDB.
This script should only be needed while we are still using the old UDB.

/home/ss/accounts/bin/expire_users

Function
Deal with expired users and expired user's homes.
  1. Put expired users into UDB Class:Transit if they are not in any class;
  2. Remove users from Transit if they are a member of another class;
  3. Remove users from groups if they are a member of Transit
  4. Move expired homes into \$FS/$EXPIRY_DIR/ after $HOME_EXPIRY_DAYS days;
  5. Delete expired homes and accounts after $RM_ACC_YRS years.

/home/ss/accounts/bin/updatesmsfiles

Function
Update schedule and enrolment files in /home/sms/sturec/data/ that are used by sms and give.

/home/sturec/TAPES/bin/get_griffin.sh

Function
Transfer NSSR5513 report set up by Geoff Whale for use by griffin

Calls:

  • /home/sturec/TAPES/bin/nss_sftp.pl -qr0 -l /home/sturec/TAPES/logs/Griffin -U2
to sftp griffin download files from echo-cs@file.sims.unsw.edu.au:outbound/and delete remote copy once downloaded.
  • /home/sturec/TAPES/bin/mv_nssreports.sh (See description at (4) above).

Dealing with staff and general records and accounts:

/home/ss/accounts/bin/fixpgalias.pl user/

Function
Check CSE users in the new UDB for upi and standard z aliases, and create acc commands that can be read by 'acc -L' to add missing upi etc.

/home/ss/accounts/bin/updatestaff.pl

Function
Update new UDB from staff allocation files in /home/teachadmin/

See separate documentation on updatestaff

/home/ss/accounts/bin/updateold staff

Function
Update old UDB from staff data and New UDB.
  • Make the same changes to the old UDB (using /home/ss/accounts/bin/updatestaff) that have already been made to the new UDB (using '/home/ss/accounts/bin/updatestaff -N')
  1. Use old acc to update the old UDB (using:
    '/home/ss/accounts/bin/updatestaff -n'.
    This creates the file: 'acc.out'.
  2. Create a list of new users in 'acc.out' that don't yet exist in the old UDB.
  3. Extract the (uid, home, and name) of these new users who should already exist in the new UDB, and add these details to 'acc.out'.
  4. Run amended 'acc.out' through old acc to make changes in the old UDB.
This script should only be needed while we are still using the old UDB.

/home/ss/bin/trusted-hosts-update

Function
Update the NIS netgroup: trusted-hosts, in the CSE PgSQL database, from data in '/home/conform/config/cse/nodeinfo'.
  • A trusted host is taken to be any machine whose network interface is defined in nodeinfo to be on one of the subnets:
    'cse-trust-*' , or 'cse-servers' .
  • Uses /home/ss/accounts/bin/mkhostgroups.pl, which uses /home/conform/bin/cffield.pl
Note: I don't know if this netgroup is in use any longer.

/home/ss/bin/nis-access-update

Function
Update the various NIS netgroups responsible for CSE host access.
These NIS netgroups are stored in the CSE UDB (Postgres) database, and are updated from user and group data taken from the same database.
  • Uses:
    1. /home/ss/accounts/bin/mkaccessgroups.pl
      perl script which creates access group tuples from specifications in /home/ss/accounts/access.d/
      These specifications are infix expressions involving UDB groups/classes evaluated using acc.
    2. /home/ss/accounts/bin/pgsql.pl
      This is a cut down version of psgl that can process PostGres commands
  • Currently used to create netgroups for 'feldman_access' and 'grieg_access'.
I don't know if these netgroups are in use any longer.

/home/ss/accounts/bin/fixprimarynames

Function
Check and fix that student primary login name is of the form z${uid}, and not of the old local personalised form generated by OW acc.
Need to check whether this still needs to be done.

/home/ss/accountd/ssh/bin/run_mkclassauth

Function
Update class account SSH keys
  • Runs:
    1. /home/ss/accounts/ssh/bin/mkclassauth.pl
      Function:
      Creates the ~/.ssh/authorized_keys file for class accounts (actually for members of Subject_Utility/ or pracexam/)
      Sends email to class account notifying them of changes or errors.
  • Logs activity in: /home/ss/accounts/ssh/log
    This file is moved aside at the end of the year.

updatestu.pl

Function
Update the UDB from student course class enrolments.
It does this by creating and processing various files in the working directory:
/home/ss/accounts/student/
  • Creates 'alloc' , the initial enrolment allocation file, from:
    1. official enrolment files in /home/sturec/DATA/, and
    2. unofficial enrolment records in /home/ss/accounts/student/unoff.comm
    Format: {regno course/program year term category}
    where: category is the course/program category that determines creation/expiry dates.
  • Creates 'expire' , the expire data file, by passing alloc though:
    /home/ss/bin/unsw_dates.pl
    Format: {regno course/program_class-[min|max] expirydate}
    where: course/program_class is the UDB class
    eg: 'COMP9331t1_Student', '3843_Student'
  • Creates 'udb' , the current UDB class file, by running an acc command to extract all user members of 'Subject_Student' or 'Course_Student'
    Format: {regno course/program_class expirydate}
  • Processeses 'udb' and 'expire' using:
    /home/ss/accounts/bin/processalloc.pl
    to create the file of acc commands: 'acc.out'.
  • Processes these acc commands through /usr/local/bin/acc.

Note:

  1. processalloc.pl is also used by updatestaff.pl, and produces output that attempts to explain why certain expiry dates were chosen. Such explanations are usually more important with staff class membership changes than for student class membership changes, as staff often have more complex reasons for their various expiries, and they often want to know why any changes were made.
  2. updatestu has a single config file: /home/ss/lib/updatestu.config, and it will strictly enforce all official and unofficial enrolments.

In general, student course/program expiry dates depend on:

  1. Enrolment (year,session) of (course/program).
  2. Mapping (course/program) -> category
    where:
    category: [ugc pgrc ugrc ngrc cp rp op phd]
    is defined in the config file.
  3. Mapping (year,session,category) -> (creation/expiry) date
    This is done by:
    /home/ss/bin/unsw_dates.pl
    which has its own config file: /home/ss/lib/unsw_dates.config specifying (creation/expiry) dates for each (category/duty/reason).

Changes to (course/program) creation/expiry dates may be made by changing any one (or both) of these two config files.

Retrieved from ‘https://techdocs.cseunsw.tech/mediawiki/index.php?title=CSE_Account_creation_and_maintenance_scripts&oldid=1150’