Changes in 3a0aa53: edited olympic-plan-for-paris-marseille-2024.md

				wiki/info/landscape/olympic-plan-for-paris-marseille-2024.md
			
          @@ -57,7 +57,29 @@ ADDITIONAL_JAVA_ARGS="${ADDITIONAL_JAVA_ARGS} -Dcom.sap.sse.debranding=true"

           Make sure to align the ``INSTALL_FROM_RELEASE`` parameter to match up with the release used on site.

          -## Switching

          +## SSH Tunnels

          +

          +The baseline is again the Tokyo 2020 set-up. Besides the jump host's re-naming from ``tokyo-ssh.sapsailing.com`` to ``paris-ssh.sapsailing.com``. The tunnel scripts for ``sap-p1-2`` that assume ``sap-p1-2`` is (primary) master seem to be faulty. At least, they don't establish a reverse port forward for port 8888 which, however, seems necessary to let cloud replicas reach the on-site master. ``sap-p1-2`` becoming (primary) on-site master means that ``sap-p1-1`` has failed. This can be a problem with the application process but could even be a hardware issue where the entire machine has crashed and has become unavailable. Therefore, ``sap-p1-2`` must take over at least the application and become primary master, and this requires the reverse port forward like this: ``-R '*:8888:localhost:8888'``

          +

          +The ports and their semantics:

          +

          +*   443: HTTPS port of security-service.sapsailing.com (or its local replacement through NGINX)

          +*  5673: Outbound RabbitMQ to use by on-site master (or local replacement)

          +*  5675: Inbound RabbitMQ for replication from security-service.sapsailing.com (or local replacement)

          +*  9443: NGINX HTTP port on sap-p1-1 (also reverse-forwarded from paris-ssh.sapsailing.com)

          +*  9444: NGINX HTTP port on sap-p1-2 (also reverse-forwarded from paris-ssh.sapsailing.com)

          +* 10201: MongoDB on sap-p1-1

          +* 10202: MongoDB on sap-p1-2

          +* 10203: MongoDB on paris-ssh.sapsailing.com

          +* 15673: HTTP to RabbitMQ administration UI of the RabbitMQ server reached on port 5673

          +* 15675: HTTP to RabbitMQ administration UI of the RabbitMQ server reached on port 5675

          +

          +### Regular Operations

          +

          +* Three MongoDB nodes form the ``paris2024`` replica set: ``sap-p1-1:10201``, ``sap-p1-2:10202``, and ``paris-ssh.sapsailing.com:10203``, where SSH tunnels forward ports 10201..10203 such that everywhere on the three hosts involved the replica set can be addressed as ``mongodb://localhost:10201,localhost:10202,localhost:10203/?replicaSet=paris2024&retryWrites=true&readPreference=nearest``

          +* ``sap-p1-1`` runs the ``paris2024`` production master from ``/home/sailing/servers/paris2024`` against local database ``paris2024:paris2024``, replicating from ``security-service.sapsailing.com`` through SSH tunnel from local port 443 pointing to ``security-service.sapsailing.com`` (which actually forwards to the ALB hosting the rules for ``security-service.sapsailing.com`` and RabbitMQ ``rabbit.internal.sapsailing.com`` tunneled through port 5675, with the RabbitMQ admin UI tunneled through port 15675; outbound replication goes to local port 5673 which tunnels to ``rabbit-eu-west-3.sapsailing.com`` whose admin UI is reached through port 15673 which tunnels to ``rabbit-eu-west-3.sapsailing.com:15672``

          +* ``sap-p1-2`` runs the ``paris2024`` shadow master from ``/home/sailing/servers/paris2024`` against local database ``paris2024:paris2024-shadow``, replicating from ``security-service.sapsailing.com`` through SSH tunnel from local port 443 pointing to ``security-service.sapsailing.com`` (which actually forwards to the ALB hosting the rules for ``security-service.sapsailing.com`` and RabbitMQ ``rabbit.internal.sapsailing.com`` tunneled through port 5675, with the RabbitMQ admin UI tunneled through port 15675; outbound replication goes to local port 5673 which tunnels to the RabbitMQ running locally on ``sap-p1-2``, port 5672 whose admin UI is then reached through port 15673 which tunnels to ``sap-p1-2:15672``

          +* The database ``mongodb://mongo0.internal.sapsailing.com,mongo1.internal.sapsailing.com/security_service?replicaSet=live`` is backed up on a regular basis (nightly) to the local MongoDB replica set ``paris2024`` DB named ``security_service``.

           ### Production Master Failure

          @@ -69,10 +91,6 @@ Approach: Switch to previous shadow master, re-configuring all SSH tunnels accor

           As in the Tokyo 2020 scenario; in particular, the local security service must be started which will work off a regularly updated local MongoDB copy of the cloud-based security-service.sapsailing.com; this also requires to adjust /etc/hosts and the tunnels accordingly.

          -## SSH Tunnels

          -

          -TBD; baseline is again the Tokyo 2020 set-up.

          -

           ## Test Plan for Test Event Marseille July 2023

           ### Test Internet Failure

          @@ -93,6 +111,10 @@ Combine the above scenarios: a failing production master (hardware or VM-only) w

           ## TODO Before / During On-Site Set-Up (Both, Test Event and OSG2024)

          +* Set up Global Accelerator and have the already established DNS record ``paris2024.sapsailing.com`` (placeholder that points to the Dynamic ALB in the default region ``eu-west-1`` to effectively forward to the central reverse proxy and ultimately the archive server's landing page) become an alias pointing to this Global Accelerator

          +* Set up logging buckets for ALBs in all supported regions

          +* Set up ALBs in all supported regions, define their three rules (redirect for ``paris2024.sapsailing.com/`` path; forward to public target group for all other ``paris2024.sapsailing.com`` traffic; default rule forwarding to IP-based target group containing the ``eu-west-1`` central reverse proxy) and register them with the Global Accelerator

           * Add SSH public keys for password-less private keys of ``sap-p1-1`` and ``sap-p1-2`` to ``ec2-user@paris-ssh.sapsailing.com:.ssh/authorized_keys.org`` so that when the authorized_keys file is updated automatically, the on-site keys are still preserved.

           * Create LetsEncrypt certificates for the NGINX installations for paris2024.sapsailing.com and security-service.sapsailing.com and install to the two on-site laptops' NGINX environments

          -* Ensure the MongoDB installations on both laptops use 

          \ No newline at end of file

          +* Ensure the MongoDB installations on both laptops use the ``paris2024`` replica set

          +* Adjust Athena queries to include all ALB logging buckets from all regions

          \ No newline at end of file

...	...	@@ -57,7 +57,29 @@ ADDITIONAL_JAVA_ARGS="${ADDITIONAL_JAVA_ARGS} -Dcom.sap.sse.debranding=true"
57	57
58	58	Make sure to align the ``INSTALL_FROM_RELEASE`` parameter to match up with the release used on site.
59	59
60		-## Switching
	60	+## SSH Tunnels
	61	+
	62	+The baseline is again the Tokyo 2020 set-up. Besides the jump host's re-naming from ``tokyo-ssh.sapsailing.com`` to ``paris-ssh.sapsailing.com``. The tunnel scripts for ``sap-p1-2`` that assume ``sap-p1-2`` is (primary) master seem to be faulty. At least, they don't establish a reverse port forward for port 8888 which, however, seems necessary to let cloud replicas reach the on-site master. ``sap-p1-2`` becoming (primary) on-site master means that ``sap-p1-1`` has failed. This can be a problem with the application process but could even be a hardware issue where the entire machine has crashed and has become unavailable. Therefore, ``sap-p1-2`` must take over at least the application and become primary master, and this requires the reverse port forward like this: ``-R '*:8888:localhost:8888'``
	63	+
	64	+The ports and their semantics:
	65	+
	66	+* 443: HTTPS port of security-service.sapsailing.com (or its local replacement through NGINX)
	67	+* 5673: Outbound RabbitMQ to use by on-site master (or local replacement)
	68	+* 5675: Inbound RabbitMQ for replication from security-service.sapsailing.com (or local replacement)
	69	+* 9443: NGINX HTTP port on sap-p1-1 (also reverse-forwarded from paris-ssh.sapsailing.com)
	70	+* 9444: NGINX HTTP port on sap-p1-2 (also reverse-forwarded from paris-ssh.sapsailing.com)
	71	+* 10201: MongoDB on sap-p1-1
	72	+* 10202: MongoDB on sap-p1-2
	73	+* 10203: MongoDB on paris-ssh.sapsailing.com
	74	+* 15673: HTTP to RabbitMQ administration UI of the RabbitMQ server reached on port 5673
	75	+* 15675: HTTP to RabbitMQ administration UI of the RabbitMQ server reached on port 5675
	76	+
	77	+### Regular Operations
	78	+
	79	+* Three MongoDB nodes form the ``paris2024`` replica set: ``sap-p1-1:10201``, ``sap-p1-2:10202``, and ``paris-ssh.sapsailing.com:10203``, where SSH tunnels forward ports 10201..10203 such that everywhere on the three hosts involved the replica set can be addressed as ``mongodb://localhost:10201,localhost:10202,localhost:10203/?replicaSet=paris2024&retryWrites=true&readPreference=nearest``
	80	+* ``sap-p1-1`` runs the ``paris2024`` production master from ``/home/sailing/servers/paris2024`` against local database ``paris2024:paris2024``, replicating from ``security-service.sapsailing.com`` through SSH tunnel from local port 443 pointing to ``security-service.sapsailing.com`` (which actually forwards to the ALB hosting the rules for ``security-service.sapsailing.com`` and RabbitMQ ``rabbit.internal.sapsailing.com`` tunneled through port 5675, with the RabbitMQ admin UI tunneled through port 15675; outbound replication goes to local port 5673 which tunnels to ``rabbit-eu-west-3.sapsailing.com`` whose admin UI is reached through port 15673 which tunnels to ``rabbit-eu-west-3.sapsailing.com:15672``
	81	+* ``sap-p1-2`` runs the ``paris2024`` shadow master from ``/home/sailing/servers/paris2024`` against local database ``paris2024:paris2024-shadow``, replicating from ``security-service.sapsailing.com`` through SSH tunnel from local port 443 pointing to ``security-service.sapsailing.com`` (which actually forwards to the ALB hosting the rules for ``security-service.sapsailing.com`` and RabbitMQ ``rabbit.internal.sapsailing.com`` tunneled through port 5675, with the RabbitMQ admin UI tunneled through port 15675; outbound replication goes to local port 5673 which tunnels to the RabbitMQ running locally on ``sap-p1-2``, port 5672 whose admin UI is then reached through port 15673 which tunnels to ``sap-p1-2:15672``
	82	+* The database ``mongodb://mongo0.internal.sapsailing.com,mongo1.internal.sapsailing.com/security_service?replicaSet=live`` is backed up on a regular basis (nightly) to the local MongoDB replica set ``paris2024`` DB named ``security_service``.
61	83
62	84	### Production Master Failure
63	85
...	...	@@ -69,10 +91,6 @@ Approach: Switch to previous shadow master, re-configuring all SSH tunnels accor
69	91
70	92	As in the Tokyo 2020 scenario; in particular, the local security service must be started which will work off a regularly updated local MongoDB copy of the cloud-based security-service.sapsailing.com; this also requires to adjust /etc/hosts and the tunnels accordingly.
71	93
72		-## SSH Tunnels
73		-
74		-TBD; baseline is again the Tokyo 2020 set-up.
75		-
76	94	## Test Plan for Test Event Marseille July 2023
77	95
78	96	### Test Internet Failure
...	...	@@ -93,6 +111,10 @@ Combine the above scenarios: a failing production master (hardware or VM-only) w
93	111
94	112	## TODO Before / During On-Site Set-Up (Both, Test Event and OSG2024)
95	113
	114	+* Set up Global Accelerator and have the already established DNS record ``paris2024.sapsailing.com`` (placeholder that points to the Dynamic ALB in the default region ``eu-west-1`` to effectively forward to the central reverse proxy and ultimately the archive server's landing page) become an alias pointing to this Global Accelerator
	115	+* Set up logging buckets for ALBs in all supported regions
	116	+* Set up ALBs in all supported regions, define their three rules (redirect for ``paris2024.sapsailing.com/`` path; forward to public target group for all other ``paris2024.sapsailing.com`` traffic; default rule forwarding to IP-based target group containing the ``eu-west-1`` central reverse proxy) and register them with the Global Accelerator
96	117	* Add SSH public keys for password-less private keys of ``sap-p1-1`` and ``sap-p1-2`` to ``ec2-user@paris-ssh.sapsailing.com:.ssh/authorized_keys.org`` so that when the authorized_keys file is updated automatically, the on-site keys are still preserved.
97	118	* Create LetsEncrypt certificates for the NGINX installations for paris2024.sapsailing.com and security-service.sapsailing.com and install to the two on-site laptops' NGINX environments
98		-* Ensure the MongoDB installations on both laptops use
...	...	\ No newline at end of file
	0	+* Ensure the MongoDB installations on both laptops use the ``paris2024`` replica set
	1	+* Adjust Athena queries to include all ALB logging buckets from all regions
...	...	\ No newline at end of file