New World gitlab server

Notes

1. In AWS, set up the EC2 instance on the sydney-hosts subnetwork with free-for-all security group and 1TB of had disk
2. After first boot, follow Initial setup of a Debian AMI instance
3. Login and set root password
4. Set /etc/hostname with the server name
5. Install the krb5-user package using apt-get and answer:
   • Realm = AD.UNSW.EDU.AU
   • Servers = ad.unsw.edu.au
   • No administration server
6. Add to /etc/dhcp/dhclient.conf (after the main stanza):
   • supersede domain-name-servers 129.94.0.196, 129.94.0.197;
   • supersede domain-search "cse.unsw.edu.au";
   • supersede domain-name "cse.unsw.edu.au";
7. Reboot
8. Check the Kerberos configuration by getting a ticket using zID with kinit, and then kdestroy to get rid of it
9. Run apt-get install ca-certificates perl
10. Follow:
    • https://about.gitlab.com/install/#debian (omit Postfix step for now)
11. Note that gitlab-ee is *not* installed on the AWS instance at time of writing. gitlab-ce, however, is installed.
12. When using the unattended-upgrades package, add the following in /etc/apt/apt.conf.d/50unattended-upgrades, under "Origin-Patterns", to allow unattended updates of gitlab as well as Debian:
    • "origin=packages.gitlab.com/gitlab/gitlab-ce,codename=${distro_codename},label=gitlab-ce"
13. Uses LetsEncrypt to get an SSL certificate for web access

Network configuration

1. The host is in AWS. It is called "nw-syd-gitlab" and is in hostlist.csv (cfengine) to reserve its internal IP address against use by any other CSE-managed host
2. This IP address is 10.197.85.252 (see also below). Due to it being in hostlist.csv, there will be an entry for it in /etc/hosts on all cfengine-managed hosts (i.e., login and VLAB servers, lab computers, etc.). Problematically, the routing required to allow it to be accessed from the outside world means that to access it from, basically, anywhere (including CSE subnetworks) users should use its public IP address or DNS name (see next) and not its internal IP address or name
3. The server has a public IP address (52.65.64.190) and a corresponding DNS entry of "nw-syd-gitlab.cseunsw.tech"
4. The server is configured with a static IP address and static routing (to allow it to talk to the outside world). DHCP needs to be disabled. See /etc/network/interfaces.d/eth0 reproduced below. Note that the default route points to AWS' gateway, not CSE's gateway/VPN-endpoint

/etc/network/interfaces.d/eth0

# Comment out the active line in the following file so
# there's only our (non-DHCP) entry for eth0 for the
# system to set up
#
# /etc/udev/rules.d/75-cloud-ifupdown.rules
#
# ----------
#
# All clients should be accessing this server via its
# public IP address (same as DNS for gitlab.cseunsw.tech)
# The routing below ensures that traffic to the AD and
# DNS servers which *this* host uses go through the CSE
# tunnel and thus appears to these servers to be from a
# UNSW-based host so the lookups give the appropriate
# result

auto eth0
allow-hotplug eth0

iface eth0 inet static
	#
	address	10.197.85.252/24
	gateway	10.197.85.1
	#
	# AD and UNSW DNS server traffic needs to go through the
	# tunnel so that the lookup is valid for a UNSW-based host
	up	/sbin/ip route add 10.116.160.0/24 via 10.197.85.5
	up	/sbin/ip route add 10.157.49.0/24 via 10.197.85.5
	up	/sbin/ip route add 131.236.3.0/24 via 10.197.85.5
	up	/sbin/ip route add 129.94.0.0/24 via 10.197.85.5

Monitoring

1. Add the snmpd package
2. See /etc/snmp/snmpd.conf (below) and copy the relevant executables from the cfengine warehouse into /usr/local/bin
3. systemctl enable snmpd and systemctl start snmpd
4. Noting the comments above about routing, monitor2 can talk to nw-syd-gitlab's SNMP daemon (snmpd) because both servers are on the same subnetwork

/etc/snmp/snmpd.conf

syslocation    AWS
syscontact     ss@cse.unsw.edu.au

rocommunity    csereader
rocommunity    vlabreader

extend         usercount /usr/local/bin/snmpd_usercount
extend         loadaverage /usr/local/bin/snmpd_loadaverage
extend         diskstats /usr/local/bin/getdiskstats
extend         chkcfengine /usr/local/bin/chkcfengine
extend         devicetemperature /usr/local/bin/get_device_temperature
extend         memoryusage /usr/local/bin/get_memory_usage

Entry in hostlist.csv

nw-syd-gitlab,10.197.85.252,monping+mon2other

Postfix configuration

Add the following to /etc/postfix/main.cf and then run systemctl reload postfix:

relayhost = [smtp.unsw.edu.au]

Note that SPF and/or DMARC DMS entries may be required to be added to the domain DNS when sending to non-UNSW addresses (e.g., gmail) if the emails are being treated as spam.

iptables rules (/etc/iptables/rules.v4)

# Generated by iptables-save v1.8.7 on Wed Jun 21 10:39:34 2023
*filter
:INPUT DROP [432515:22169635]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [229402354:1713993119625]
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -s 129.94.0.0/16 -i eth0 -p tcp -m multiport --dports 22,80,443 -j ACCEPT
-A INPUT -s 149.171.0.0/16 -i eth0 -p tcp -m multiport --dports 22,80,443 -j ACCEPT
-A INPUT -s 10.0.0.0/8 -i eth0 -p tcp -m multiport --dports 22,80,443 -j ACCEPT
-A INPUT -s 10.197.85.0/24 -i eth0 -p udp -m udp --dport 161 -j ACCEPT
-A INPUT -i eth0 -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -i eth0 -p tcp -m multiport --dports 22,80,443 -j ACCEPT
COMMIT
# Completed on Wed Jun 21 10:39:34 2023

Backup notes

From: Mei
Subject: Re: Cron <root@nw-k17-login1> /gitlabbackup/bin/backup.sh /home; /gitlabbackup/bin/backup.sh /var/opt/gitlab/backups
To: Peter
Date: 22jun2023

Hi Peter,
Thank you for your suggestion.

I think I got the scripts working on cron now. Will start them at the correct times tomorrow.

Created gitlabbackup user on both cse and nw-syd-gitlab using the same uid.

The backup starts by rsyncing /etc/gitlab, /home and moving "gitlab-backup create" tar file into staging folder (/home/gitlabbackup/BACKUPS) on nw-syd-gitlab. In the staging folder, modify owner to gitlabbackup recursively, no change to file permission tho.

Then from nw-syd-login1, rsync the content of the staging folder into /gitlabbackup/BACKUPS. Run as gitlabbackup user. 3 backups kept. Use hard links where possible with --link-dest option 

Regards,
Mei Cheng