60a38b7433186d86336991e6a24b6422cf9eccfb
wiki/info/landscape/paris2024/olympic-setup.md
| ... | ... | @@ -176,8 +176,11 @@ For the server side on paris-ssh and on the both P1s the following parameters ha |
| 176 | 176 | ``` |
| 177 | 177 | ClientAliveInterval 3 |
| 178 | 178 | ClientAliveCountMax 3 |
| 179 | +GatewayPorts yes |
|
| 179 | 180 | ``` |
| 180 | 181 | |
| 182 | +The ``GatewayPorts`` directive is required in order to get port forwards (including reverse port forwards) accept the "*" as bind address to bind to 0.0.0.0 instead of 127.0.0.1. |
|
| 183 | + |
|
| 181 | 184 | ExitOnForwardFailure will force ssh to exit if one of the port forwards fails. ConnectTimeout manages the time in seconds until an initial connection fails. AliveInterval (client and server) manages the time in seconds after ssh/sshd are sending client and server alive probes. CountMax is the number of retries for those probes. |
| 182 | 185 | |
| 183 | 186 | The settings have been verified by executing a network change on both the laptops, the ssh tunnel returns after a couple of seconds. |
| ... | ... | @@ -475,7 +478,7 @@ In order to have a local copy of the ``security_service`` database, a CRON job e |
| 475 | 478 | |
| 476 | 479 | ### Master |
| 477 | 480 | |
| 478 | -The master configuration is described in ``/home/sailing/servers/master/master.conf`` and can be used to produce a clean set-up like this: |
|
| 481 | +The master configuration on ``sap-p1-1`` is described in ``/home/sailing/servers/master/master.conf`` and can be used to produce a clean set-up like this: |
|
| 479 | 482 | |
| 480 | 483 | ``` |
| 481 | 484 | rm env.sh; cat master.conf | ./refreshInstance.sh auto-install-from-stdin |
| ... | ... | @@ -486,7 +489,7 @@ If the laptops cannot reach ``https://releases.sapsailing.com`` due to connectiv |
| 486 | 489 | This way, a clean new ``env.sh`` file will be produced from the config file, including the download and installation of a release. The ``master.conf`` file looks approximately like this: |
| 487 | 490 | |
| 488 | 491 | ``` |
| 489 | -INSTALL_FROM_RELEASE=build-202106012325 |
|
| 492 | +INSTALL_FROM_RELEASE=build-202306271444 |
|
| 490 | 493 | SERVER_NAME=paris2024 |
| 491 | 494 | MONGODB_URI="mongodb://localhost:10201,localhost:10202,localhost:10203/${SERVER_NAME}?replicaSet=paris2024&retryWrites=true&readPreference=nearest" |
| 492 | 495 | # RabbitMQ in eu-west-1 (rabbit.internal.sapsailing.com) is expected to be found through SSH tunnel on localhost:5675 |
| ... | ... | @@ -500,9 +503,38 @@ REPLICATE_MASTER_BEARER_TOKEN="***" |
| 500 | 503 | REPLICATION_HOST=localhost |
| 501 | 504 | REPLICATION_PORT=5673 |
| 502 | 505 | USE_ENVIRONMENT=live-master-server |
| 503 | -ADDITIONAL_JAVA_ARGS="${ADDITIONAL_JAVA_ARGS} -Dcom.sap.sse.debranding=true" |
|
| 506 | +ADDITIONAL_JAVA_ARGS="${ADDITIONAL_JAVA_ARGS} -Dcom.sap.sse.debranding=true -Dpolardata.source.url=https://www.sapsailing.com:22443 -Dwindestimation.source.url=https://www.sapsailing.com:22443" |
|
| 507 | +``` |
|
| 508 | + |
|
| 509 | +### Secondary Master |
|
| 510 | + |
|
| 511 | +The secondary master configuration on ``sap-p1-2`` can be used to fail over quickly if the primary master on ``sap-p1-1`` fails for some reason. The configuration is described in ``/home/sailing/servers/secondary_master/secondary_master.conf`` and can be used to produce a clean set-up like this: |
|
| 512 | + |
|
| 513 | +``` |
|
| 514 | + rm env.sh; cat secondary_master.conf | ./refreshInstance.sh auto-install-from-stdin |
|
| 515 | +``` |
|
| 516 | + |
|
| 517 | +This way, a clean new ``env.sh`` file will be produced from the config file, including the download and installation of a release. The ``secondary_master.conf`` file looks approximately like this: |
|
| 518 | + |
|
| 519 | +``` |
|
| 520 | +INSTALL_FROM_RELEASE=build-202306271444 |
|
| 521 | +SERVER_NAME=paris2024 |
|
| 522 | +MONGODB_URI="mongodb://sap-p1-1:27017,sap-p1-2:27017/${SERVER_NAME}?replicaSet=security_service&retryWrites=true&readPreference=nearest" |
|
| 523 | +# RabbitMQ in eu-west-1 (rabbit.internal.sapsailing.com) is expected to be found through SSH tunnel on localhost:5675 |
|
| 524 | +# Replication of shared services from central security-service.sapsailing.com through SSH tunnel 443:security-service.sapsailing.com:443 |
|
| 525 | +# with a local /etc/hosts entry mapping security-service.sapsailing.com to 127.0.0.1 |
|
| 526 | +REPLICATE_MASTER_QUEUE_HOST=localhost |
|
| 527 | +REPLICATE_MASTER_QUEUE_PORT=5675 |
|
| 528 | +REPLICATE_MASTER_BEARER_TOKEN="***" |
|
| 529 | +# Outbound replication to RabbitMQ through SSH tunnel with port forward on port 5673, regularly to RabbitMQ on localhost, |
|
| 530 | +# can be re-mapped to the cloud RabbitMQ running on rabbit-eu-west-3.internal.sapsailing.com to make this the "primary" master |
|
| 531 | +REPLICATION_HOST=localhost |
|
| 532 | +REPLICATION_PORT=5673 |
|
| 533 | +USE_ENVIRONMENT=live-master-server |
|
| 534 | +ADDITIONAL_JAVA_ARGS="${ADDITIONAL_JAVA_ARGS} -Dcom.sap.sse.debranding=true -Dpolardata.source.url=https://www.sapsailing.com:22443 -Dwindestimation.source.url=https://www.sapsailing.com:22443" |
|
| 504 | 535 | ``` |
| 505 | 536 | |
| 537 | + |
|
| 506 | 538 | ### Replicas |
| 507 | 539 | |
| 508 | 540 | The on-site replica on ``sap-p1-2`` can be configured with a ``replica.conf`` file in ``/home/sailing/servers/replica``, using |
| ... | ... | @@ -516,7 +548,7 @@ The file looks like this: |
| 516 | 548 | ``` |
| 517 | 549 | # Regular operations; sap-p1-2 replicates sap-p1-1 using the rabbit-eu-west-3.sapsailing.com RabbitMQ in the cloud through SSH tunnel. |
| 518 | 550 | # Outbound replication, though not expected to become active, goes to a local RabbitMQ |
| 519 | -INSTALL_FROM_RELEASE=build-202106012325 |
|
| 551 | +INSTALL_FROM_RELEASE=build-202306271444 |
|
| 520 | 552 | SERVER_NAME=paris2024 |
| 521 | 553 | MONGODB_URI="mongodb://localhost:10201,localhost:10202,localhost:10203/${SERVER_NAME}-replica?replicaSet=paris2024&retryWrites=true&readPreference=nearest" |
| 522 | 554 | # RabbitMQ in eu-west-3 is expected to be found locally on port 5673 |
| ... | ... | @@ -533,29 +565,12 @@ USE_ENVIRONMENT=live-replica-server |
| 533 | 565 | ADDITIONAL_JAVA_ARGS="${ADDITIONAL_JAVA_ARGS} -Dcom.sap.sse.debranding=true" |
| 534 | 566 | ``` |
| 535 | 567 | |
| 536 | -Replicas in region ``eu-west-1`` can be launched using the following user data, making use of the established MongoDB live replica set in the region: |
|
| 537 | - |
|
| 538 | -``` |
|
| 539 | -INSTALL_FROM_RELEASE=build-202106012325 |
|
| 540 | -SERVER_NAME=paris2024 |
|
| 541 | -MONGODB_URI="mongodb://mongo0.internal.sapsailing.com,mongo1.internal.sapsailing.com,dbserver.internal.sapsailing.com:10203/paris2024-replica?replicaSet=live&retryWrites=true&readPreference=nearest" |
|
| 542 | -USE_ENVIRONMENT=live-replica-server |
|
| 543 | -REPLICATION_CHANNEL=paris2024-replica |
|
| 544 | -REPLICATION_HOST=rabbit-eu-west-3.sapsailing.com |
|
| 545 | -REPLICATE_MASTER_SERVLET_HOST=paris-ssh.internal.sapsailing.com |
|
| 546 | -REPLICATE_MASTER_SERVLET_PORT=8888 |
|
| 547 | -REPLICATE_MASTER_EXCHANGE_NAME=paris2024 |
|
| 548 | -REPLICATE_MASTER_QUEUE_HOST=rabbit-eu-west-3.sapsailing.com |
|
| 549 | -REPLICATE_MASTER_BEARER_TOKEN="***" |
|
| 550 | -ADDITIONAL_JAVA_ARGS="${ADDITIONAL_JAVA_ARGS} -Dcom.sap.sse.debranding=true" |
|
| 551 | -``` |
|
| 552 | - |
|
| 553 | 568 | (Adjust the release accordingly, of course). (NOTE: During the first production days of the event we noticed that it was really a BAD IDEA to have all replicas use the same DB set-up, all writing to the MongoDB PRIMARY of the "live" replica set in eu-west-1. With tens of replicas running concurrently, this led to a massive block-up based on MongoDB not writing fast enough. This gave rise to a new application server AMI which now has a MongoDB set-up included, using "replica" as the MongoDB replica set name. Now, each replica hence can write into its own MongoDB instance, isolated from all others and scaling linearly.) |
| 554 | 569 | |
| 555 | 570 | In other regions, instead an instance-local MongoDB shall be used for each replica, not interfering with each other or with other databases: |
| 556 | 571 | |
| 557 | 572 | ``` |
| 558 | -INSTALL_FROM_RELEASE=build-202106012325 |
|
| 573 | +INSTALL_FROM_RELEASE=build-202306271444 |
|
| 559 | 574 | SERVER_NAME=paris2024 |
| 560 | 575 | MONGODB_URI="mongodb://localhost/paris2024-replica?replicaSet=replica&retryWrites=true&readPreference=nearest" |
| 561 | 576 | USE_ENVIRONMENT=live-replica-server |