FinanceServersMigrationPlan

Overview

This page explains the necessary steps to follow to fully update timesheets, finance, hr and tenders VMs. These communicate with each other and other VMs through direct postgresql connections and also by Rabbit AMQP messages. They are high profile and require down time for users when this procedure is done, so it needs proper preparation before starting to limit the down time and to make sure they will work correctly after migration with minimal impact on users.

Steps before down time

• Create the VMs according to their needs
• Install all packages required
• Copy all scripts on finance and install the same cron jobs
• Add sudoers command aliases on finance:

# Cmnd alias specification
Cmnd_Alias FINANCESCRIPT= /home/finance/synchCommand.sh . /home/finance/midnightScripts.sh /home/finance/.financeLock
Cmnd_Alias PULLTSSCRIPT= /home/finance/synchCommand.sh . /home/finance/pullTimeSheets.sh * /home/finance/.financeLock
Cmnd_Alias SLIPEXPSCRIPT= /home/finance/synchCommand.sh . /home/finance/slipExpAndDatedProjects.sh /home/finance/.financeLock

# User www-data needs to run the above command as finance
www-data ALL=(finance) NOPASSWD:FINANCESCRIPT
www-data ALL=(finance) NOPASSWD:PULLTSSCRIPT
www-data ALL=(finance) NOPASSWD:SLIPEXPSCRIPT

• Take full DB dumps from the old VMs
• update all DB functions to use domain names when accessing remote DBs instead of IPs. Former IPs and what they should become:

10.0.10.48 finance.thehumanjourney.net
10.0.10.23 timesheets.thehumanjourney.net
10.0.10.42 hrdb.thehumanjourney.net
10.0.10.63 tenders.thehumanjourney.net
10.0.10.130 archives-db.thehumanjourney.net
192.168.98.24 mapdata.thehumanjourney.net

• Update passwords where necessary
• Update IPs to DNS names and also passwords in all config.php files on any of the VMs
• Restore dumps to new VMs
• Edit /etc/hosts and add names and IPs to all VMs involved so they can be addressed by DNS rather than IPs:

10.0.10.49 finance.thehumanjourney.net
192.168.98.51 timesheets.thehumanjourney.net
10.0.10.43 hrdb.thehumanjourney.net
10.0.10.180 tenders.thehumanjourney.net
10.0.10.130 archives-db.thehumanjourney.net
192.168.98.24 mapdata.thehumanjourney.net

• Open port 5432 on the firewall (IP restricted) for timesheets to access finance
• Open port 5432 on the firewall (IP restricted) for HRDB to access timesheets
• Open port 5432 on the firewall (IP restricted) for tenders to access mapdata
• Install and configure AMQP client on Finance according to this guide on OA wiki
• Test "pull time sheets" script on PMDB interface - this will test both ways finance - timesheets connection
• Test making changes to staff on PMDB interface - this will test HRDB connection to timesheets, admin, finance and also RabbitMQ sending messages from Finance to archives
• Test making changes to projects on PMDB interface - this will test finance RabbitMQ sending messages to archives
• Test making changes to an episode on Tenders interface and see if the changes were transmitted to mapdata into tenders table on oasites DB
• Test sending e-mails from timesheets and from finance
• Test database backup procedures worked correctly

Steps during down time

• Take DB dumps for all old VMs
• Restore only data to new VMs
• Stop DB engine and apache on all old VMs to prevent any external connections still trying the old VMs
• Update DNS records to point to new VMs

Steps after down time

• Test all interfaces work well
• Test special functions that might no longer work
  • Make staff changes in HRDB and check if the change was propagated to TRS, Finance, Admin DB on Finance VM, Tenders, Archives
  • Make project changes on Finance and see if the change was propagated to Archives - this checks if RabbitMQ worked well
• Monitor any support tickets related to any of these services and first find a cause in the migration done.