CSE Account creation and maintenance scripts

The CSE User DataBase (usually referred to as the UDB), is a PostGres database that stores the account details of all CSE users. It is usually accessed and modified using the acc command, although it can also be accessed and modified directly (with appropriate access permissions) using the postgres command pgsql.

This document describes the main system scripts that are used to update the UDB from official student enrolment records, or from staff allocation and HR records. Most of the scripts are located somewhere in /home/ss, usually in /home/ss/accounts/bin/, but many (particularly those related to student downloads and official copies of enrolment records) are also located in '/home/sturec/TAPES/bin/.

UDBupdate

Location

/home/ss/accounts/bin/UDBupdate

Function

This is the main script that updates CSE student and staff accounts. It ties together and calls most of the other account related scripts.

• Currently run nightly on the OW machine:synth by the maint system:

  synth:/usr/local/maint/nightly/S10_NSSupdate ->

  /home/ss/accounts/bin/NSSupdate ->

  UDBupdate

Note that the maint system is called at 2am on synth by /usr/local/maint/run nightly which is run by cron.

• Its activity is logged in: /home/ss/logs/NSSupdate.$MONTH (eg: NSSupdate.Dec).

  The log records when each constituant script below is called, and captures their output.

• UDBupdate, and most of the scripts called by it, may be run with:
  • The '-h' option, which produces specific and more detailed help information related to the script.
  • The '-n' option, which reports any changes to be made, but does not actually make any.
• Most of the scripts run by UDBupdate may be run independently of UDBupdate. This is often useful when debugging problems.

UDBupdate should be unhooked from the maint system in the Old world.

What follows is a description of the scripts called by UDBupdate, in the order in which they are called.

---

Dealing with student records and accounts:

tape.schedule

Location

/home/ss/accounts/bin/tape.schedule

Function

List the SiMs (Student Information Management System) files expected to be downloaded from New South Solutions (NSS) today.

NSS is the old name for the SiMs unit in Central IT that deals with downloading Student Enrolments and Records for the various schools around UNSW. Various enrolment downloads may be obtained after NSS runs their so-called batch jobs in the evening or overnight. These jobs are identified by NSSR Batch-number (eg: NSSR3502).

This script:

• Needs to be updated whenever the parameters of the NSS download jobs are changed.
• Contains comments detailing how and where to log in to change the various NSS download parameters (these comments are particularly useful).

  Note: Download parameters need to be updated manually, usually sometime before the start of each session, to reflect the shifting set of term enrollments that we need to obtain.

• Records the day of week, or day of month, when each download job is expected to be run. Each job line should also include the details of the job's download parameters.

  Once upon a long time ago, many of these batch jobs were only run weekly or monthly. All of them are now run daily.

• UDBupdate will compare the list of downloads actually obtained, against the list of downloads expected (the list produced by this script), and will send email to the mail alias updates@cse.unsw.edu.au if there are any differences.

---

nss_sftp.pl

Location

/home/sturec/TAPES/bin/nss_sftp.pl

Function

perl script that uses sftp to copy NSS downloads to /home/sturec/TAPES/

• There are a large number of different options available which can be used to set various values to the (local and remote) (machines, users and directories). Use the '-h' option for more details.
• This script used to have to be run on the CSE machine synth, because file.sims.unsw.edu.au (the remote file staging site) would not accept ssh connections from any other CSE machine.
  This restriction needs to be confirmed.
• Currently, synth is an OW conformed machine. When this is changed to a NW machine, you will need to make sure that the required perl modules are installed (Net::SFTP::Foreign and Fcntl ':mode' )

---

mv_nssreports.sh

Location

/home/sturec/TAPES/bin/mv_nssreports.sh

Function

Give downloaded NSS data files standard names (in /home/sturec/TAPES) that are used/known elsewhere in CSE (by sms, teachadmin, other (ss) scripts).

Names are of the form:

YYTT_SCHOOL.TYPE

where:

  YY     year (last two digits of year);
  TT     term (T[0123] | H[1-6])
  SCHOOL Organisational unit 'owning' the records (eg: COMPSC,ENG,MULTI).
  TYPE   Download records type:
     enr     NSSR3502 Enrolment records
     STDNT   NSSR5046 Student records
     SUBJ    NSSR5046 Subject records
     sched   NSSR2922 Schedule records
     griff   NSSR5513 Griffin records

mv_nssreports.sh uses:

• /home/sturec/bin/extract.pl

  perl script which extracts data fields from standard downloads given field specifications (specified in /home/sturec/lib/field.*)

• /home/ss/bin/unsw_dates.pl

  perl script generating dates for UNSW term start,end,expiry,etc.

---

mksturecdata

Location

/home/sturec/bin/mksturecdata

Function

Generate a single form of program or course enrolment data (generally one line per student course or program enrolment), from the different forms of download files obtained by nss_sftp.pl and renamed in mv_nssreports.sh.

These files are kept in:

/home/sturec/DATA/CCYYTT.(program|course)[.full]

Note: Many standard download files contain the same data in different formats. This script was written in an attempt to create single enrolment files with a uniform and consistent format, with data taken from the many different download files.

---

updatestu.pl

Location

/home/ss/accounts/bin/updatestu.pl

Function

Update the student records in the CSE UDB from enrolment records in /home/sturec/DATA/.

This script is described in much more detail in the separate documentation below.

---

updateold stu

Location

/home/ss/accounts/bin/updateold

Function

Update student records in CSE's old UDB with reference to student records in CSE's new UDB.

• Make the same changes to the old UDB (using '/home/ss/accounts/bin/updatestu')
  that have already been made to the new UDB (with '/home/ss/accounts/bin/updatestu -N').
  1. Use /home/ss/accounts/bin/updatestu -n to use the old acc to update the old UDB from student records

     This creates the file: 'acc.out' , but does not run it through acc to make any changes yet.

  2. Create a list of new users in 'acc.out.new' that don't yet exist in the old UDB.

     Note: Although the users don't yet exist in the old UDB, they should already exist in the new UDB because updatestu -N was just run before this.

  3. Extract the (uid, home, and name) of these new users from the new UDB, and add these details to 'acc.out'.
  4. Run amended 'acc.out' through old acc to make changes to the old UDB.

This script should only be needed while we are still using the old UDB.

---

expire_users

Location

/home/ss/accounts/bin/expire_users

Function

Deal with expired users and expired user's homes.

1. Put expired users into UDB Class:Transit if they are not in any class;
2. Remove users from Transit if they are a member of another class;
3. Remove users from groups if they are a member of Transit
4. Move expired homes into \$FileSystem/.expired/ after $HOME_EXPIRY_DAYS days;
5. Delete expired homes and accounts after $RM_ACC_YRS years.

---

updatesmsfiles

Location

/home/ss/accounts/bin/updatesmsfiles

Function

Update schedule and enrolment files in /home/sms/sturec/data/ that are used by sms and give.

---

get_griffin.sh

Location

/home/sturec/TAPES/bin/get_griffin.sh

Function

Transfer and rename NSSR5513 reports set up by Geoff Whale for use by griffin

Calls

• /home/sturec/TAPES/bin/nss_sftp.pl -qr0 -l /home/sturec/TAPES/logs/Griffin -U2

to sftp griffin download files from echo-cs@file.sims.unsw.edu.au:outbound/ and delete remote copy once downloaded.

• /home/sturec/TAPES/bin/mv_nssreports.sh

to rename the downloaded report (See previous description).

Note

The Griffin files need to have been downloaded and renamed before they are subsequently copied and processed by cron jobs set up for the nss account on the machine rautavaara (See rautavaara:/var/spool/cron/crontabs/nss).
The Griffin system is a legacy system set up by the academic Geoff Whale (now retired) to render some general aspects of UNSW course data more accessible and useful to course administrators. Apart from the minimal support we provide here, most of the Griffin data and processing lies outside of CSG's purview.

---

Dealing with staff accounts, netgroups and SSH access:

fixpgalias.pl user/

Location

/home/ss/accounts/bin/fixpgalias.pl

Function

Check CSE users in the new UDB for upi and standard z aliases, and create acc commands that can be read by 'acc -L' to add missing upi etc.

---

updatestaff.pl

Location

/home/ss/accounts/bin/updatestaff.pl

Function

Update new UDB from staff allocation files in /home/teachadmin/

See the separate documentation below

---

updateold staff

Location

/home/ss/accounts/bin/updateold

Function

Update old UDB from staff data and New UDB.

• Make the same changes to the old UDB (using /home/ss/accounts/bin/updatestaff) that have already been made to the new UDB (using '/home/ss/accounts/bin/updatestaff -N')

1. Use old acc to update the old UDB (using:

   '/home/ss/accounts/bin/updatestaff -n'.

   This creates the file: 'acc.out'.

2. Create a list of new users in 'acc.out' that don't yet exist in the old UDB.
3. Extract the (uid, home, and name) of these new users who should already exist in the new UDB, and add these details to 'acc.out'.
4. Run amended 'acc.out' through old acc to make changes in the old UDB.

This script should only be needed while we are still using the old UDB.

---

trusted-hosts-update

Location

/home/ss/bin/trusted-hosts-update

Function

Update the NIS netgroup: trusted-hosts, in the CSE PgSQL database, from data in '/home/conform/config/cse/nodeinfo'.

• A trusted host is taken to be any machine whose network interface is defined in nodeinfo to be on one of the subnets:

  'cse-trust-*' , or 'cse-servers' .

• Uses /home/ss/accounts/bin/mkhostgroups.pl, which uses /home/conform/bin/cffield.pl

Note: I don't know if this netgroup is in use any longer.

---

nis-access-update

Location

/home/ss/bin/nis-access-update

Function

Update the various NIS netgroups responsible for CSE host access.
These NIS netgroups are stored in the CSE UDB (Postgres) database, and are updated from user and group data taken from the same database.

• Uses:
  1. /home/ss/accounts/bin/mkaccessgroups.pl

     perl script which creates access group tuples from specifications in /home/ss/accounts/access.d/

     These specifications are infix expressions involving UDB groups/classes evaluated using acc.

  2. /home/ss/accounts/bin/pgsql.pl

     This is a cut down version of psgl that can process PostGres commands

• Currently used to create netgroups for 'feldman_access' and 'grieg_access'.

I don't know if these netgroups are in use any longer.

---

fixprimarynames

Location

/home/ss/accounts/bin/fixprimarynames

Function

Check and fix that student primary login name is of the form z${uid}, and not of the old local personalised form generated by OW acc.

Need to check whether this still needs to be done.

---

run_mkclassauth

Location

/home/ss/accountd/ssh/bin/run_mkclassauth

Function

Update class account SSH keys.

• Run: /home/ss/accounts/ssh/bin/mkclassauth.pl

This is the main script that creates the file: ~/.ssh/authorized_keys for all class account members of Subject_Utility/ or pracexam/.

Email is sent to class accounts and users notifying them of changes or errors.

Once a week, email is also sent to class accounts warning them of users included without an expiry date.

Activity is logged in: /home/ss/accounts/ssh/log

• The log file is moved aside by run_mkclassauth at the end of the year.

---

Updatestu

Location

/home/ss/accounts/bin/updatestu -> updatestu.pl

Function

Update the UDB from student course class enrolment summaries stored in /home/sturec/DATA/. It does this by creating and processing various files in the working directory:

/home/ss/accounts/student/

• Creates 'alloc' , the initial enrolment allocation file, from:
  1. Summary files of official enrolment data in /home/sturec/DATA/, and
  2. Unofficial enrolment records in /home/ss/accounts/student/unoff.comm

  Format: { regno course/program year term category }

  where:

  year term

  Are determined from the enrolment file

  category

  Is the course/program category that determines creation/expiry dates.

  This is set in the config file /home/ss/lib/updatestu.config

• Creates 'expire' , the expire data file, by passing alloc though:

  /home/ss/bin/unsw_dates.pl

  Format: { regno course/program_class-[min|max] expirydate }

  where:

  course/program_class

  Is the UDB class (eg: 'COMP9331t1_Student' , '3843_Student'

• Creates 'udb' , the current UDB class file, by running acc to extract all user members of 'Subject_Student/' or 'Course_Student/'

  Format: { regno course/program_class expirydate }

• Creates 'acc.out' , the file of acc commands, by running 'udb' and 'expire' through:

  /home/ss/accounts/bin/processalloc.pl

  processalloc.pl is also used by updatestaff.pl, and produces output that attempts to explain why certain expiry dates were chosen. Such explanations are usually more important with staff class membership changes than for student class membership changes, as staff often have more complex reasons for their various expiries, and they often want to know why any changes were made.

• Processes 'acc.out' by passing it through acc.

Config File

The config file: /home/ss/lib/updatestu.config defines:

1. The location of the various source files for:
   • Official student course and program enrolments;
   • Unofficial enrolment records;
   • Personal details.
2. What script or function to call to process each type of source file.
3. The students requiring a CSE account and account class. Students are deemed to require an account if they are enrolled in:
   • Plans defined by specific majors
   • Programs defined by specific program numbers;
   • Courses defined by either:

   • The School that runs the course, or
   • Specific course numbers, or

   • Not enrolled in specifically excluded courses
4. The mapping of course to course category, and program to program category.

   These categories affect the creation and expiry dates of the corresponding course or program account classes.

   • Course categories generally end in 'c' and include: ugc gdc pgrc ugrc ngrc,
   • Program categories generally end in a 'p' and include: cp rp op phd.

   The categories are defined in more detail in the config file.

Creation and Expiry Dates

In general, the date that:

1. The student is put into a course or program account class (called its creation date), or
2. The student expires from the course or program account class (called its expiry date),

depends on:

1. The Year and session of enrolment;
2. The course or program category.

The mapping of course or program to category is defined in the config file (above).

The mapping of (year,session,category) to (creation/expiry) date is done by:

/home/ss/bin/unsw_dates.pl

which has its own config file: /home/ss/lib/unsw_dates.config specifying (creation/expiry) dates for each (category/duty/reason).

Global changes to (course/program) creation/expiry dates (ones affecting all students) may be made by changing any one (or both) of the config files:

1. /home/ss/lib/unsw_dates.config, or
2. /home/ss/lib/updatestu.config.

Individual changes to (course/program) creation/expiry dates (ones affecting single students) may be made by making entries in:

/home/ss/accounts/student/unoff.comm.