89f7d0e3ced65c2b1609a1a0517f70da9f72e01c
wiki/info/landscape/creating-ec2-image-for-webserver-from-scratch.md
| ... | ... | @@ -169,6 +169,28 @@ maxretry = 5 |
| 169 | 169 | * Ensure that fail2ban will be started automatically when the instance starts: `chkconfig --level 23 fail2ban on` and start it right away with `service fail2ban start`. You can see which filters are active using `service fail2ban status`. |
| 170 | 170 | * Ensure you have EC2 / EBS snapshot backups for the volumes by tagging them as follows: ``WeeklySailingInfrastructureBackup=Yes`` for ``/var/www/static``, ``/var/log``, ``/var/log/old`` and ``/var/log/old/cache``, ``DailySailingBackup=Yes`` for ``/home``. |
| 171 | 171 | |
| 172 | +## Automating archive failover |
|
| 173 | + |
|
| 174 | +We have a script in our git repo called `switchoverArchive.sh`, which takes a path to the macros file and two timeout values (in seconds). It checks the macros file and checks if the following lines are present: |
|
| 175 | + |
|
| 176 | +``` |
|
| 177 | +Define ARCHIVE_IP 172.31.7.12 |
|
| 178 | +Define ARCHIVE_FAILOVER_IP 172.31.43.140 |
|
| 179 | +Define PRODUCTION_ARCHIVE ${ARCHIVE_IP} |
|
| 180 | +``` |
|
| 181 | +Then it curls the primary/main archive's `/gwt/status` (with the first timeout value) and, if healthy, sets the production value to the definition of the archive; however, if unhealthy, a |
|
| 182 | +second curl occurs (with the second timeout value) and if this again returns unhealthy then the production value above is this time set to be the value of the failover definition. |
|
| 183 | +After these changes, key admins are notified and the apache config is reloaded. This only happens though if the new value differs from the currently known value: |
|
| 184 | +ie. if already healthy, and the health checks pass, then no reload or email occurs. |
|
| 185 | +To install, enter `crontab -e`; set the frequency to say `* * * * *`; add the path to the script; parameterise it with the path to the macros file, the first timeout value and the second timeout value (both seconds); and then |
|
| 186 | +write and quit, to install the cronjob. |
|
| 187 | + |
|
| 188 | +``` |
|
| 189 | +# Example crontab |
|
| 190 | +* * * * * /home/wiki/gitwiki/configuration/switchoverArchive.sh "/etc/httpd/conf.d/000-macros.conf" 2 9 |
|
| 191 | +``` |
|
| 192 | + |
|
| 193 | +If you want to quickly run this script, consider installing it in /usr/local/bin, via `ln -s TARGET_PATH LINK_NAME`, in that directory. |
|
| 172 | 194 | |
| 173 | 195 | ## Basic setup for reverse proxy instance |
| 174 | 196 |