CSE Account creation and maintenance scripts

The CSE User DataBase (usually referred to as the UDB), is a PostGres database that stores the account details of all CSE users. It is usually accessed and modified using the acc command, although it can also be accessed and modified directly (with appropriate access permissions) using the postgres command pgsql.

This document describes the main system scripts that are used to update the UDB from official student enrolment records, or from staff allocation and HR records. Most of the scripts are located somewhere in /home/ss, usually in /home/ss/accounts/bin/, but many (particularly those related to student downloads and official copies of enrolment records) are also located in '/home/sturec/TAPES/bin/.

This document also includes a description of some of the manual processes and procedures that someone from the CSG will need to go through to update and/or set up some of the Data files and download files required by the account creation and maintenance scripts. These are described in the section below on Data Files

UDBupdate

Location

/home/ss/accounts/bin/UDBupdate

Function

This is the main script that updates CSE student and staff accounts. It ties together and calls most of the other account related scripts.

• Currently run nightly on the OW machine:synth by the maint system:

  synth:/usr/local/maint/nightly/S10_NSSupdate ->

  /home/ss/accounts/bin/NSSupdate ->

  UDBupdate

Note that the maint system is called at 2am on synth by /usr/local/maint/run nightly which is run by cron.

• Its activity is logged in: /home/ss/logs/NSSupdate.$MONTH (eg: NSSupdate.Dec).

  The log records when each constituant script below is called, and captures their output.

• UDBupdate, and most of the scripts called by it, may be run with:
  • The '-h' option, which produces specific and more detailed help information related to the script.
  • The '-n' option, which reports any changes to be made, but does not actually make any.
• Most of the scripts run by UDBupdate may be run independently of UDBupdate. This is often useful when debugging problems.

UDBupdate should be unhooked from the maint system in the Old world.

What follows is a description of the scripts called by UDBupdate, in the order in which they are called.

---

Dealing with student records and accounts:

Student downloads and updates are processed before staff ones are.

batchreminder

Location

/home/ss/accounts/bin/batchreminder

Function

• Send an email to the mail alias: batchupdate reminding the associated users to update the NSS batch job parameters that produce the CSE enrolment downloads.
• This reminder is sent during the one week period before the course creation date (ie: the earliest date that students can be put into their ${Course}_Student UDB class).
• As Graduate Diploma courses operate according to online term dates (as opposed to standard UNSW term dates), this script tests for and sends a separate reminder to change the GLDIP course download parameters.

tape.schedule

Location

/home/ss/accounts/bin/tape.schedule

Called by

UDBupdate

Function

Lists the SiMs (Student Information Management System) files expected to be downloaded from New South Solutions (NSS) today.

Details

NSS is the old name for the SiMs unit in Central IT that deals with downloading Student Enrolments and Records for the various schools around UNSW. Various enrolment downloads are obtained by nss_sftp.pl after NSS runs their so-called batch jobs in the evening or overnight. These jobs are identified by their NSSR Batch-number (eg: NSSR3502).

This script is essentially a here list containing every NSS download batch job that is run by CSE at SiMs. Each line in the list records:

• The day of week, or day of month, when the job is expected to be run.

  Note: Once upon a long time ago, many of these batch jobs were only run weekly or monthly. All of them are now run daily.

• The job's NSSR Batch-number or job name (eg: NSSR3502).
• The job's download parameters. Note that the NSS download job parameters need to be updated within SiMS manually, usually sometime before the start of each session, to reflect the shifting set of term enrolments that we need to obtain. To help remind us to update these parameters, the script: batchreminder is also run by UDBupdate, which sends a email to specific CSG users to update these parameters at the right time.
• The name that the downloaded file should be given once it has been transferred to CSE. This name usually includes the year and term, and will usually change with the change of job parameters.

When run, this script evaluates when each job should be run, and produces a list of files that are expected to be created by downloads run today. When UDBupdate runs this command, it compares the list of downloads expected (ie: the list produced by this script) against the list of downloads actually obtained from SiMs, and will send email to the mail alias: updates@cse.unsw.edu.au if there are any differences.

This script also contains copious comments detailing how and where to log in to SiMS to change the various NSS download parameters. These comments are particularly useful, as they are currently not documented anywhere else.

---

nss_sftp.pl

Location

/home/sturec/TAPES/bin/nss_sftp.pl

Function

This is a perl script that uses sftp to copy NSS downloads into /home/sturec/TAPES/

• There are a large number of different options available which can be used to assign various known values to the (local and remote) (machines, users and directories) expected to be used by sftp. Use the '-h' option for more details.
• This script has to be run on the CSE machine synth, because file.sims.unsw.edu.au (the remote file staging site) does not accept ssh connections from any other CSE machine. This restriction needs to be confirmed.
• To work, this script requires the perl modules: Net::SFTP::Foreign and Fcntl ':mode' to be installed. Currently, synth is an OW conformed machine, but when this is changed to a NW machine, you will need to ensure that these perl modules are installed there too.

---

mv_nssreports.sh

Location

/home/sturec/TAPES/bin/mv_nssreports.sh

Function

Give downloaded NSS data files standard names (in /home/sturec/TAPES) that are used/known elsewhere in CSE (by sms, teachadmin, other (ss) scripts).

Names are of the form:

YYTT_SCHOOL.TYPE

where:

  YY     year (last two digits of year);
  TT     term (T[0123] | H[1-6])
  SCHOOL Organisational unit 'owning' the records (eg: COMPSC,ENG,MULTI).
  TYPE   Download records type:
     enr     NSSR3502 Enrolment records
     STDNT   NSSR5046 Student records
     SUBJ    NSSR5046 Subject records
     sched   NSSR2922 Schedule records
     griff   NSSR5513 Griffin records

mv_nssreports.sh uses:

• /home/sturec/bin/extract.pl

  perl script which extracts data fields from standard downloads given field specifications (specified in /home/sturec/lib/field.*)

• /home/ss/bin/unsw_dates.pl

  perl script generating dates for UNSW term start,end,expiry,etc.

---

mksturecdata

Location

/home/sturec/bin/mksturecdata

Function

Generate a single form of program or course enrolment data (generally one line per student course or program enrolment), from the different forms of download files obtained by nss_sftp.pl and renamed in mv_nssreports.sh.

These files are kept in:

/home/sturec/DATA/CCYYTT.(program|course)[.full]

Note: Many standard download files contain the same data in different formats. This script was written in an attempt to create single enrolment files with a uniform and consistent format, with data taken from the many different download files.

---

updatestu.pl

Location

/home/ss/accounts/bin/updatestu.pl

Function

Update the student records in the CSE UDB from enrolment records in /home/sturec/DATA/.

This script is described in much more detail in the following updatestu section.

---

updateold stu

Location

/home/ss/accounts/bin/updateold

Function

Update student records in CSE's old UDB with reference to student records in CSE's new UDB.

• Make the same changes to the old UDB (using '/home/ss/accounts/bin/updatestu')
  that have already been made to the new UDB (with '/home/ss/accounts/bin/updatestu -N').
  1. Use /home/ss/accounts/bin/updatestu -n to use the old acc to update the old UDB from student records

     This creates the file: 'acc.out' , but does not run it through acc to make any changes yet.

  2. Create a list of new users in 'acc.out.new' that don't yet exist in the old UDB.

     Note: Although the users don't yet exist in the old UDB, they should already exist in the new UDB because updatestu -N was just run before this.

  3. Extract the (uid, home, and name) of these new users from the new UDB, and add these details to 'acc.out'.
  4. Run amended 'acc.out' through old acc to make changes to the old UDB.

This script should only be needed while we are still using the old UDB.

---

expire_users

Location

/home/ss/accounts/bin/expire_users

Function

Deal with expired users and expired user's homes.

1. Put expired users into UDB Class:Transit if they are not in any class;
2. Remove users from Transit if they are a member of another class;
3. Remove users from groups if they are a member of Transit
4. Move expired homes into \$FileSystem/.expired/ after $HOME_EXPIRY_DAYS days;
5. Delete expired homes and accounts after $RM_ACC_YRS years.

---

updatesmsfiles

Location

/home/ss/accounts/bin/updatesmsfiles

Function

Update schedule and enrolment files in /home/sms/sturec/data/ that are used by sms and give.

---

get_griffin.sh

Location

/home/sturec/TAPES/bin/get_griffin.sh

Function

Transfer and rename NSSR5513 reports set up by Geoff Whale for use by griffin

Calls

• /home/sturec/TAPES/bin/nss_sftp.pl -qr0 -l /home/sturec/TAPES/logs/Griffin -U2

to sftp griffin download files from echo-cs@file.sims.unsw.edu.au:outbound/ and delete remote copy once downloaded.

• /home/sturec/TAPES/bin/mv_nssreports.sh

to rename the downloaded report (See previous description).

Note

The Griffin files need to have been downloaded and renamed before they are subsequently copied and processed by cron jobs set up for the nss account on the machine rautavaara (See rautavaara:/var/spool/cron/crontabs/nss).
The Griffin system is a legacy system set up by the academic Geoff Whale (now retired) to render some general aspects of UNSW course data more accessible and useful to course administrators. Apart from the minimal support we provide here, most of the Griffin data and processing lies outside of CSG's purview.

---

Dealing with staff accounts, netgroups and SSH access:

fixpgalias.pl user/

Location

/home/ss/accounts/bin/fixpgalias.pl

Function

Check CSE users in the new UDB for upi and standard z aliases, and create acc commands that can be read by 'acc -L' to add missing upi etc.

---

updatestaff.pl

Location

/home/ss/accounts/bin/updatestaff.pl

Function

Update new UDB from staff allocation files in /home/teachadmin/

See the separate documentation below

---

updateold staff

Location

/home/ss/accounts/bin/updateold

Function

Update old UDB from staff data and New UDB.

• Make the same changes to the old UDB (using /home/ss/accounts/bin/updatestaff) that have already been made to the new UDB (using '/home/ss/accounts/bin/updatestaff -N')

1. Use old acc to update the old UDB (using:

   '/home/ss/accounts/bin/updatestaff -n'.

   This creates the file: 'acc.out'.

2. Create a list of new users in 'acc.out' that don't yet exist in the old UDB.
3. Extract the (uid, home, and name) of these new users who should already exist in the new UDB, and add these details to 'acc.out'.
4. Run amended 'acc.out' through old acc to make changes in the old UDB.

This script should only be needed while we are still using the old UDB.

---

trusted-hosts-update

Location

/home/ss/bin/trusted-hosts-update

Function

Update the NIS netgroup: trusted-hosts, in the CSE PgSQL database, from data in '/home/conform/config/cse/nodeinfo'.

• A trusted host is taken to be any machine whose network interface is defined in nodeinfo to be on one of the subnets:

  'cse-trust-*' , or 'cse-servers' .

• Uses /home/ss/accounts/bin/mkhostgroups.pl, which uses /home/conform/bin/cffield.pl

Note: I don't know if this netgroup is in use any longer.

---

nis-access-update

Location

/home/ss/bin/nis-access-update

Function

Update the various NIS netgroups responsible for CSE host access.
These NIS netgroups are stored in the CSE UDB (Postgres) database, and are updated from user and group data taken from the same database.

• Uses:
  1. /home/ss/accounts/bin/mkaccessgroups.pl

     perl script which creates access group tuples from specifications in /home/ss/accounts/access.d/

     These specifications are infix expressions involving UDB groups/classes evaluated using acc.

  2. /home/ss/accounts/bin/pgsql.pl

     This is a cut down version of psgl that can process PostGres commands

• Currently used to create netgroups for 'feldman_access' and 'grieg_access'.

I don't know if these netgroups are in use any longer.

---

fixprimarynames

Location

/home/ss/accounts/bin/fixprimarynames

Function

Check and fix that student primary login name is of the form z${uid}, and not of the old local personalised form generated by OW acc.

Need to check whether this still needs to be done.

---

run_mkclassauth

Location

/home/ss/accountd/ssh/bin/run_mkclassauth

Function

Update class account SSH keys.

• Run: /home/ss/accounts/ssh/bin/mkclassauth.pl

This is the main script that creates the file: ~/.ssh/authorized_keys for all class account members of Subject_Utility/ or pracexam/.

Email is sent to class accounts and users notifying them of changes or errors.

Once a week, email is also sent to class accounts warning them of users included without an expiry date.

Activity is logged in: /home/ss/accounts/ssh/log

• The log file is moved aside by run_mkclassauth at the end of the year.

---

Updatestu

Location

/home/ss/accounts/bin/updatestu -> updatestu.pl

Function

Update the UDB from student course class enrolment summaries stored in /home/sturec/DATA/. It does this by creating and processing a number of files in the working directory: /home/ss/accounts/student/. These files are:

• 'alloc' - the initial enrolment allocation file, created from:

1. Summary files of official enrolment data in /home/sturec/DATA/, and
2. Unofficial enrolment records in /home/ss/accounts/student/unoff.comm

File format: { regno course/program year term category }

where:

year term

Are determined from the enrolment file

category

Is the course/program category that determines creation/expiry dates.

This is set in the config file /home/ss/lib/updatestu.config

• 'expire' - the expire data file, created by passing alloc though:

/home/ss/bin/unsw_dates.pl

File format: { regno course/program_class-[min|max] expirydate }

where:

course/program_class

Is the UDB class (eg: 'COMP9331t1_Student' , '3843_Student' )

• 'udb' - the current UDB class file, created by running acc to extract all user members of 'Subject_Student/' or 'Course_Student/'

File format: { regno course/program_class expirydate }

• 'acc.out' - the file of acc commands, created by running 'udb' and 'expire' through:

/home/ss/accounts/bin/processalloc.pl

processalloc.pl compares the existing class memberships in 'udb' with the intended memberships in 'expire' and creates the file of acc commands that will turn the existing into the intended. processalloc.pl is also used by updatestaff.pl, and produces output that attempts to explain why certain expiry dates were chosen. Such explanations are usually more important with staff class membership changes than for student class membership changes, as staff often have more complex reasons for their various expiries, and they often want to know why any changes were made.

• 'acc.out' is finally processed by passing it through acc. However, before it is processed, updatestu compares the number of additions and deletions in acc.out, against the expected number of changes to be made today. The expected number of changes to be made on any day, will vary depending on the current time of year with respect to the start or end of terms. ie: At the so called 'creation date' , we expect to add large numbers of new course or program enrolments to the UDB, while at the 'expiry date' , we expect to remove large numbers of course or program enrolments from the UDB. Of course, we always expect some changes (additions and deletions) even outside of these special times (because many students enrol late and discontinue early) and an upper estimate of these numbers is put into the file: updatestu.config. In any case, updatestu processes the alloc file through unsw_dates.pl with appropriate arguments to determine the number of additions or deletions expected to have to be made today. If the current number of changes in acc.out exceeds the upper bound of the total number of changes expected today, then none of the changes in acc.out are processed unless updatestu is run with the force option (-F). This prevents UDBupdate (which runs updatestu) from erroneously making changes that may be entirely due to a problem with either the official enrolment records or with UDB access. System staff should exclude these possibilities before attempting to rerun updatestu with -F.

Config File

The config file: /home/ss/lib/updatestu.config defines:

1. The location of the various source files for:
   • Official student course and program enrolments;
   • Unofficial enrolment records;
   • Personal details.
2. What script or function to call to process each type of source file.
3. The students requiring a CSE account and account class. Students are deemed to require an account if they are enrolled in:
   • Plans defined by specific majors
   • Programs defined by specific program numbers;
   • Courses defined by either:

   • The School that runs the course, or
   • Specific course numbers, or

   • Not enrolled in specifically excluded courses
4. The mapping of course to course category, and program to program category.

   These categories affect the creation and expiry dates of the corresponding course or program account classes.

   • Course categories generally end in 'c' and include: ugc gdc pgrc ugrc ngrc,
   • Program categories generally end in a 'p' and include: cp rp op phd.

   The categories are defined in more detail in the config file.

Creation and Expiry Dates

In general, the date that:

1. The student is put into a course or program account class (called its creation date), or
2. The student expires from the course or program account class (called its expiry date),

depends on:

1. The Year and session of enrolment;
2. The course or program category.

The mapping of course or program to category is defined in the config file (above).

The mapping of (year,session,category) to (creation/expiry) date is done by:

/home/ss/bin/unsw_dates.pl

which has its own config file: /home/ss/lib/unsw_dates.config specifying (creation/expiry) dates for each (category/duty/reason).

Global changes to (course/program) creation/expiry dates (ones affecting all students) may be made by changing any one (or both) of the config files:

1. /home/ss/lib/unsw_dates.config, or
2. /home/ss/lib/updatestu.config.

Individual changes to (course/program) creation/expiry dates (ones affecting single students) may be made by making entries in:

/home/ss/accounts/student/unoff.comm.

Data Files

There are essentially four types of data files that are used and inspected when creating or updating CSE accounts:

1. Official Student course enrolment data.
   • These enrolment data files record the official enrolments of students in courses, programs and plans.
   • These data files are produced daily by batch jobs run by SiMs on their infrastructure. They are downloaded every morning to /home/sturec/TAPES/ using nss_sftp.pl.
   • These SiMs batch jobs select the data to be downloaded according to NSS job parameters that we need to periodically update throughout the year - typically a week before students are expected to be put into their enrolled course UDB classes (called the Class Creation Date), which is generally a few weeks before the start of each session. The script: batchreminder is run by UDBupdate, and a week before the class creation date, batchreminder sends an email reminder to the mail alias: batchupdate to remind the associated CSG employee(s) to update the NSS enrolment batch parameters. Once the parameters have been updated, and tape.schedule has been modified, batchreminder stops sending email to batchupdate.
   • The procedure that needs to be followed to update the batch job parameters is documented in the script: tape.schedule. The script itself should be edited to record the parameters changed, and the enrolment download files that are expected to be obtained as a result of these parameter changes.
2. Official Staff course allocation data.
   • These course allocation data files record the official allocations of CSE staff to various course teaching roles. This includes the allocation of Course Tutors, Lecturers, Administrators, Supervisors, etc, and may include allocations of casual and/or non-academic staff (eg: students employed as tutors)
   • This data is generally stored in various text files in various directories in /home/teachadmin/, and is continuously and periodically updated by the academic(s) in charge of administering academic course allocations within CSE, particularly around the start of the each session.
   • Whenever it is run, the script: updatestaff:

   1. Reads its config file defining the location of its term based staff allocation files;
   2. Looks for and reads the staff allocation files extracting the association between staff and course.

   Note:If updatestaff does not find any of the files that it is told to expect, then it produces an error message in its logs, but continues regardless.

3. Official Staff UNSW HR data.

   This download is currently only run by UNSW HR once a week (each Wednesday), after which they send an email reminder to a specific CSG employee (currently zain@cse.unsw.edu.au), notifying this person that the download job has been run, and including a link to a page providing access to the download. The CSG employee then:

   • Logs into the PiMs system using the link provided, using their unipass. Note: The employee obviously needs to have appropriate access to PiMs.
   • Transfers the file to /home/$USERNAME/Download/UNSW_COMPSC_WEEKLY_DATA_*.xlsx
   • Runs (as root): /home/ss/accounts/bin/mv_staffdata.sh $USERNAME, which:

   • Moves the .xlsx file from: /home/$USERNAME/Download/ to: /home/ss/accounts/lib/,
   • Runs: libreoffice --convert-to csv to convert the .xlsx file into a .csv file that can be read by updatestaff.

   Ideally, this procedure should be automated far more than it is, and should be made much less dependent on specific individuals within CSG/CSE manually transferring and processing the HR file.

4. Unofficial student or staff data.

   Occasionally, we need to modify the official association between some CSE student and/or staff member and a CSE course and/or program. This will typically involve removing, extending, or truncating the CSE user's membership or expiry in one or more UDB course class.

   In keeping with the two main types of CSE users (student and staff) , there are two unofficial allocation files:

   1. /home/ss/accounts/student/unoff.comm
   2. /home/ss/accounts/staff/manual.comm