Proposed home directory architecture, configuration and management
Preamble
The way it was
In the Old World the majority of the CSE home directories were stored on six main NFS servers, each server having local RAID disk storage for the home directories.
These servers had names such as ravel, kamen, etc. Home directories were located in ext3/ext4/xfs file systems created on local logical block devices and mounted under /export (e.g., /export/kamen/1, /export/ravel/2, etc.) from whence it was exported via the NFS to lab computers, VLAB servers and login servers. For ease of management and speed of fsck on reboot, each physical server had its available disk storage divided into multiple logical block devices, each of less than 1TB.
User home directory paths would be hard-coded to be on a particular file system on a particular server, e.g., /export/kamen/1/plinich.
This arrangement meant that a problem on a particular server could potentially affect 1/6th of all accounts until such time as the server itself was fixed, or until the affected home directories were restored from backup to another server and the users' home directory locations were updated. This arrangement also meant that home directories were stored in one geographic location in one of six physical servers.
The way it might be
The proposed architecture disassociates home directories from specific physical/virtual NFS servers and places home directories in virtual hard disk block devices which are actually stored either in the Amazon AWS S3 cloud, or in CSE's distributed Ceph storage cluster.
The key ideas are:
- Home directories are stored on block devices defined and existing either in Amazon's AWS S3 storage, or in CSE's own Ceph storage cluster.
- NFS servers may be physical or virtual and DO NOT have local storage for home directories. They do, however, have their own IP addresses which have NO ASSOCIATION with any of the home directory block devices. Ihe IP address of an NFS server is used solely to manage the NFS server itself and has no role in making available the home directory NFS exports.
- Home directory storage is attached to the particular NFS servers which export it via either iSCSI, in the case of AWS, or as Ceph RADOS Block Devices (RBD). Both of these attachment methods use TCP/IP and allow the home directory storage to then be mounted under Linux as a normal ext3/ext4/xfs file system.
- Each home directory storage block device is associated with a same-named
/etc/hostsentry with its own IP address. E.g., "home02" would be associated with a host entry of "home02.cse.unsw.edu.au" or just "home02" with a static IP address of, say, 129.94.242.ABC. One (and only one) home directory block storage device is associated with a single IP address.
Importantly, the NFS servers have no implicit or explicit association with any particular home directory stores. Instead, an NFS server can export an arbitrary number of home directory file systems[1].
The procedure for making a home directory's block device contents available via NFS
- Select the particular home directory block device in either AWS or Ceph. E.g., "homeXX"
- If homeXX is in AWS:
- Associate the "homeXX" block device in AWS with one of the on-site AWS Storage Gateways using the AWS web console, and make it available to local hosts via iSCSI. See ???
- Attach the homeXX block device via iSCSI to the NFS server you want to export that block device's file system.
- If homeXX is in the Ceph storage cluster:
- Map the home directory RBD into the NFS server.
- If homeXX is in AWS:
mountthe attached device on the NFS server as/export/homeXX/1[2].- Add an entry to
/etc/exportsfor the home directory file system and then runexportfs -rato tell the NFS server deamon to make the file system available. - Find the IP address associated with the home directory file system (homeXX.cse.unsw.edu.au, etc.) and add this address to the server's network interface with
ip addr add 129.94.242.ABC/24 dev eth0(check this!)
Most steps above can be automated/scripted.
Repeat the above for each home directory file system/block device.
At this point, NFS clients should be able to mount the home directory file system(s) like this (or equivalent by the automounter):
# mount -t nfs homeXX:/export/homeXX/1 /mnt
hostlist.csv
There is an entry in hostlist.csv for each home directory block device to create the associated host entries in /etc/hosts. E.g.:
… nw-syd-vxdb2,10.197.85.53,trusted+localstorage+nfs4+monping+mondisk+vlabserver+hostbasedauthtest+mon2vlab nw-syd-dbserver,10.197.85.54,trusted+nfs4+vlabserver+hostbasedauthtest+localstorage+projectwebserver+mon2vlab … homeXX,129.94.242.ABC homeYY,129.94.242.DEF … storage00,129.94.242.95,mon2storecomp storage01,129.94.242.96,mon2storecomp …
AWS Storage Gateway
The AWS Storage Gateway is an appliance that is installed on-site on either a virtual or physical server. It provides local iSCSI access to block devices defined in AWS' S3 storage. Using local SSD (ideally) storage, the Storage Gateway provides fast local access to cached data which is replicated to S3 following writes.
When the appliance is installed, the user/owner does basic configuration on the server's console to give it an IP address and to authenticate it to AWS, and from then on AWS takes over management and configuration (the latter of which is done via the AWS web console).
The screen snapshots to the right show the AWS console configuration interface for the Storage Gateway itself, and the volumes made available via iSCSI.
Note that the last snapshot, under "Connecting your volume" includes a link entitled "Connecting your volumes to Linux".
Note also that it is possible to move volumes from one AWS Storage Gateway to another without loss of data. This is most useful in case a Storage Gateway, or its underlying hardware, fails or needs to be moved. This works because all Storage Gateway data is stored/replicated in S3 in real time and can be reloaded therefrom to the new Storage Gateway's cache.
Note: Although there is an iSCSI configuration file under /etc/iscsi it is usually not edited by hand. Instead, the iscsiadm command is used to "discover" iSCSI devices and populate the configuration file, as well as set up security and to attach (login) the iSCSI device so it appears under /dev, and especially so it is automatically (re-)attached on reboot.
In the below, you'll also see how /etc/fstab can be set up so mounting of an iSCSI colume happens automatically at boot.
180 apt list | fgrep -i iscsi 181 apt-file list libiscsi-bin 182 apt-get install open-iscsi 183 cd /etc/iscsi/ 185 less iscsid.conf 186 iscsiadm -h 187 iscsiadm --mode discovery --op update --type sendtargets --portal 129.94.242.253 188 cd /etc/iscsi 192 iscsi status 193 systemctl status iscsid 196 man iscsiadm 197 sudo /sbin/iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login 198 man iscsiadm 199 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login 201 cd iscsi 209 systemctl restart iscsid 210 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login 212 systemctl restart iscsid 213 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login 214 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login -v 215 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login -h 217 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login 220 systemctl restart iscsid 223 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --logout 225 vi /etc/iscsi/nodes/iqn.1997-05.com.amazon\:homeaa/129.94.242.253\,3260\,1/default 226 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login 227 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --logout 228 systemctl restart iscsid 229 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login 230 vi /etc/iscsi/nodes/iqn.1997-05.com.amazon\:homeaa/129.94.242.253\,3260\,1/default 232 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login 233 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homebb --portal 129.94.242.253:3260,1 --login 234 vi /etc/iscsi/nodes/iqn.1997-05.com.amazon\:homebb/129.94.242.253\,3260\,1/default 235 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homebb --portal 129.94.242.253:3260,1 --login 237 mkfs.ext4 -j /dev/disk/by-path/ip-129.94.242.253\:3260-iscsi-iqn.1997-05.com.amazon\:homeaa-lun-0 238 mkfs.ext4 -j /dev/disk/by-path/ip-129.94.242.253\:3260-iscsi-iqn.1997-05.com.amazon\:homebb-lun-0 239 blkid /dev/disk/by-path/ip-129.94.242.253\:3260-iscsi-iqn.1997-05.com.amazon\:homebb-lun-0 240 blkid /dev/disk/by-path/ip-129.94.242.253\:3260-iscsi-iqn.1997-05.com.amazon\:homebb-lun-0 >> /etc/fstab 241 blkid /dev/disk/by-path/ip-129.94.242.253\:3260-iscsi-iqn.1997-05.com.amazon\:homeaa-lun-0 >> /etc/fstab 263 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homebb --portal 129.94.242.253:3260,1 --login 264 iscsiadm --mode node --targetname iqn.1997-05.com.amazon:homeaa --portal 129.94.242.253:3260,1 --login 267 vi /etc/iscsi/nodes/iqn.1997-05.com.amazon\:homebb/129.94.242.253\,3260\,1/default 271 vi /etc/iscsi/nodes/iqn.1997-05.com.amazon\:homeaa/129.94.242.253\,3260\,1/default
Notes
- ↑ Constrained, of course, by the limitation that only ONE NFS server at a time can attach a storage block device and, consequently, any single home directory file system can be exported by only one NFS server at a time. This constraint has nothing to do with the architecture being discussed but is due to a file system being mountable by only one host at a time without corruption occurring.
- ↑ This will always be mounted under "…/1" to maintain compatibility with Old World heuristics.