wiki/info/landscape/archive-server-upgrade.md
... ...
@@ -4,7 +4,7 @@
4 4
5 5
- Optionally, to accelerate DB reads, launch MongoDB replica for ``archive`` replica set (see [here](https://security-service.sapsailing.com/gwt/AdminConsole.html#LandscapeManagementPlace:)); wait until MongoDB replica is in ``SECONDARY`` state
6 6
- Launch "more like this" based on existing primary archive, adjusting the ``INSTALL_FROM_RELEASE`` user data entry to the release of choice and the ``Name`` tag to "SL Archive (New Candidate)" and adjusting the availability zone (AZ) such that it does not equal the AZ of the current production ARCHIVE and ideally has either the central reverse proxy or a disposable reverse proxy in that AZ so that cross-AZ traffic is avoided.
7
-- Wait until the new instance is done with its background tasks and CPU utilization goes to 0% (approximately 36h)
7
+- Wait until the new instance is done with its background tasks and CPU utilization goes to 0% (approximately 48h)
8 8
- Create an entry in the reverse proxy's ``/etc/httpd/conf.d/001-events.conf`` file like this:
9 9
```
10 10
Use Plain archive-candidate.sapsailing.com 172.31.46.203 8888
... ...
@@ -17,19 +17,32 @@ with ``172.31.46.203`` being an example of the internal IP address your new arch
17 17
git push
18 18
```
19 19
This will disseminate the configuration to all reverse proxies through a Git hook, merge as required and check out the appropriate configuration for that environment again, then reload the configuration in ``httpd``.
20
-- Compare server contents, either with ``compareServers`` script or through REST API, and fix any differences
20
+- Compare server contents, either with ``compareServers`` script and fix any differences
21 21
```
22 22
java/target/compareServers -ael https://www.sapsailing.com https://archive-candidate.sapsailing.com
23 23
```
24
-- Do some spot checks on the new instance
25
-- Switch reverse proxy, by adjusting the archive IP definitions at the top of ``root@sapsailing.com:/etc/httpd/conf.d/000-macros.conf``, followed again by
24
+which uses the REST API to run a full comparison, showing the differences in JSON format; and/or
25
+```
26
+ java/target/compareServers -el https://www.sapsailing.com https://archive-candidate.sapsailing.com
27
+```
28
+which runs a rather "classic" comparison of the leaderboard groups' JSON representations, showing regular "diff" output and stopping at the first leaderboard group that has differences. After having fixed the differences, you can continue with the previously differing leaderboard group using the ``-c`` option, like this:
29
+```
30
+ java/target/compareServers -cel https://www.sapsailing.com https://archive-candidate.sapsailing.com
31
+```
32
+- Do some spot checks on the new instance using the ``https://archive-candidate.sapsailing.com`` domain. Note that some links may lead back to the production instance ``www.sapsailing.com`` and replace ``www`` by ``archive-candidate`` accordingly in the URL in these cases.
33
+- Switch reverse proxy, by adjusting the archive IP definitions at the top of ``root@sapsailing.com:/etc/httpd/conf.d/000-macros.conf``. The file contains two lines close to its top, looking like this:
34
+```
35
+Define ARCHIVE_IP 172.31.42.246
36
+Define ARCHIVE_FAILOVER_IP 172.31.45.7
37
+```
38
+Comment out or delete the ``ARCHIVE_FAILOVER_IP`` line. Change ``ARCHIVE_IP`` into ``ARCHIVE_FAILOVER_IP`` to "demote" the current production ARCHIVE to being the failover instance. Insert a new ``ARCHIVE_IP`` definition with the internal IP address of the new archive server. Then, to check and then activate the changes, again:
26 39
```
27 40
httpd -t # ensure you get "OK" as the response
28 41
git checkout main
29 42
git commit -a
30 43
git push
31 44
```
32
-- Terminate old fail-over EC2 instance; you will have to disabel its termination protection first.
45
+- Terminate old fail-over EC2 instance; you will have to disable its termination protection first.
33 46
- Adjust Name tags for what is now the fail-over and what is now the primary archive server in EC2 console
34 47
35 48
[[_TOC_]]
... ...
@@ -79,7 +92,7 @@ But note: having launched the loading of all races doesn't make the new archive
79 92
```
80 93
INFO: Thread[MarkPassingCalculator for race R14 initialization,4,main]: Timeout waiting for future task com.sap.sse.util.impl.ThreadPoolAwareFutureTask@39f29211 (retrying); scheduled with executor com.sap.sse.util.impl.NamedTracingScheduledThreadPoolExecutor@51517746[Running, pool size = 7, active threads = 7, queued tasks = 352521, completed tasks = 610095][name=Default background executor]
81 94
```
82
-that will keep repeating. Watch out for the ``queued tasks`` count. It should be decreasing, and when done it should go down to 0 eventually.
95
+that will keep repeating. Watch out for the ``queued tasks`` count. It should be decreasing, and when done it should go down to 0 eventually, although you may not see a log entry with "queued tasks = 0" necessarily.
83 96
84 97
### Create a Temporary Mapping in ``/etc/httpd/conf.d/001-events.conf`` to Make New Server Accessible Before Switching
85 98
... ...
@@ -89,7 +102,7 @@ Grab the internal IP address of your freshly launched archive server candidate (
89 102
Use Plain archive-candidate.sapsailing.com 172.31.35.213 8888
90 103
```
91 104
92
-in the file ``root@sapsailing.com:/etc/httpd/conf.d/001-events.conf``, preferably towards the top of the file where it can be quickly found. Save the changes and check the configuration using the ``apachectl configtest`` command. It should give an output saying ``Syntax OK``. Only in this case reload the configuration by issuing the ``service httpd reload`` command as user ``root``. After this command has completed, you can watch your archive server candidate start up at [https://archive-candidate.sapsailing.com/gwt/status](https://archive-candidate.sapsailing.com/gwt/status) and make any changes necessary when the ``compareServers`` script (see below) notifies you of any differences that need handling.
105
+in the file ``root@sapsailing.com:/etc/httpd/conf.d/001-events.conf``, preferably towards the top of the file where it can be quickly found. Save the changes and check the configuration using the ``apachectl configtest`` or the equivalend ``httpd -t`` command. It should give an output saying ``Syntax OK``. Only in this case reload the configuration by issuing the ``service httpd reload`` command as user ``root``. After this command has completed, you can watch your archive server candidate start up at [https://archive-candidate.sapsailing.com/gwt/status](https://archive-candidate.sapsailing.com/gwt/status) and make any changes necessary when the ``compareServers`` script (see below) notifies you of any differences that need handling.
93 106
94 107
### Comparing Contents with Primary ARCHIVE Server
95 108