configuration/httpd/conf.d/switchoverArchive.sh
... ...
@@ -1,6 +1,5 @@
1 1
#!/bin/bash
2 2
MACROS_PATH=$1
3
-EMAIL=$2
4 3
TIMEOUT1=2
5 4
TIMEOUT2=9
6 5
#Purpose: Script is used to switch to the failover archive if the primary is unhealthy by altering the macros
... ...
@@ -20,7 +19,7 @@ else
20 19
alreadyHealthy=0
21 20
logger -t archive "currently unhealthy"
22 21
fi
23
-#Sets the production value to point to the variable defining the main archive IP.
22
+#Sets the production value to point to the variable defining the main archive IP, provided it isn't already set.
24 23
setProductionMainIfNotSet() {
25 24
if [[ $alreadyHealthy -eq 0 ]]
26 25
then
... ...
@@ -28,13 +27,7 @@ setProductionMainIfNotSet() {
28 27
#set production to archive
29 28
logger -t archive "Healthy: setting production to main archive"
30 29
sed -i -E "s/Define PRODUCTION .*/Define PRODUCTION \${ARCHIVE_IP}/" ${MACROS_PATH}
31
- systemctl reload httpd
32
- {
33
- echo "To: ${EMAIL}"
34
- echo Subject: Healthy
35
- echo
36
- echo Healthy: main archive online
37
- } | /usr/sbin/sendmail -t
30
+ notify "Healthy: main archive online"
38 31
else
39 32
#If already healthy then no reload or notification occurs.
40 33
logger -t archive "Healthy: already set, no change needed"
... ...
@@ -56,12 +49,7 @@ then
56 49
sed -i -E "s/Define PRODUCTION .*/Define PRODUCTION \${ARCHIVE_FAILOVER_IP}/" ${MACROS_PATH}
57 50
logger -t archive "Unhealthy: second check failed, switching to failover"
58 51
systemctl reload httpd
59
- {
60
- echo "To: ${EMAIL}"
61
- echo Subject: Unhealthy
62
- echo
63
- echo Unhealthy: main archive offline
64
- } | /usr/sbin/sendmail -t
52
+ notify-operators "Unhealthy: main archive offline"
65 53
else
66 54
logger -t archive "Unhealthy: second check still fails, failover already in use"
67 55
fi
... ...
@@ -71,4 +59,3 @@ then
71 59
else
72 60
setProductionMainIfNotSet
73 61
fi
74
-cat ${MACROS_PATH}
wiki/info/landscape/archive-server-upgrade.md
... ...
@@ -110,7 +110,7 @@ Define ARCHIVE_FAILOVER_IP xxx.xx.xx.xxx
110 110
Use Rewrite ${ARCHIVE_IP} 8888
111 111
</Macro>
112 112
```
113
-In the past we changed this directly within the ArchiveRewrite macro, but this was slow if we needed to switchover. As an improvement, which happened to also be neater, we added variables -- defined at the top -- including a variable for an up-and-running failover. In the case of an outage, we could comment the current archive and rename the failover (and then reload). This way we could also switch back if the primary returns to healthy. However, we have worked on an automation script, which now changes the PRODUCTION value to point to the variables ARCHIVE_IP or ARCHIVE_FAILOVER_IP.
113
+In the past we changed this directly within the ArchiveRewrite macro, but this was slow if we needed to switchover. As an improvement, which happened to also be neater, we added variables -- defined at the top -- including a variable for an up-and-running failover. In the case of an outage, we could comment the current archive and rename the failover (and then reload). This way we could also switch back if the primary returns to healthy. However, we have worked on an automation script, which now changes the PRODUCTION value to point to the variables ARCHIVE_IP or ARCHIVE_FAILOVER_IP. Upon switching, it calls notify-operators which can be found in /usr/local/bin as a symbollic link pointing to configuration/on-site-scripts/paris2024/notify-operators.
114 114
```
115 115
Define ARCHIVE_IP xxx.xx.xx.xxx
116 116
Define ARCHIVE_FAILOVER_IP xxx.xx.xx.xxx
... ...
@@ -120,7 +120,7 @@ Define PRODUCTION ${ARCHIVE_IP}
120 120
Use Rewrite ${PRODUCTION} 8888
121 121
</Macro>
122 122
```
123
-The script can be found in the git at **switchoverArchive.sh**. This script has 1 parameter which is the path to the macros file, containing the above macros (currently in /etc/httpd/conf.d/000-macros.conf. Run ```crontab -e``` to edit the cronjobs and add
123
+The script can be found in the git at **switchoverArchive.sh**. This script has 1 parameter which is the path to the macros file, containing the above macros (currently in /etc/httpd/conf.d/000-macros.conf). Run ```crontab -e``` to edit the cronjobs and add
124 124
```
125 125
* * * * * switchoverArchive "/etc/httpd/conf.d/000-macros.conf"
126 126
```
... ...
@@ -128,6 +128,7 @@ Then exit the editor.
128 128
To setup switchoverArchive as a command for easy usage run ``` ln -s /home/wiki/gitwiki/configuration/switchoverArchive.sh switchoverArchive```. This creates a symbolic link and makes the command accessible anywhere.
129 129
130 130
Check that the new archive service is now active, e.g., by looking at [sapsailing.com/gwt/status](https://sapsailing.com/gwt/status). It should reflect the new release in its ``release`` field.
131
+
131 132
## Tests
132 133
133 134
1. Healthy -> Stay healthy