Merge branch 'master' of ssh://sapsailing.com/home/trac/git into bug5938

+Import-Package: software.amazon.awssdk.regions

+ String RABBIT_IN_DEFAULT_REGION_HOSTNAME = "rabbit.internal.sapsailing.com";

+

+ String DEFAULT_REGION = "eu-west-1";

+

+ /**

+ * We maintain a DNS entry for "rabbit.internal.sapsailing.com" (see {@link #RABBIT_IN_DEFAULT_REGION_HOSTNAME}) in this region

+ */

+ String REGION_WITH_RABBITMQ_DNS_HOSTNAME = DEFAULT_REGION;

+

+ String REGION_WITH_DEFAULT_LOAD_BALANCER = DEFAULT_REGION;

+

+ /**

+ * Tag name used to identify instances on which a RabbitMQ installation is running. The tag value is currently interpreted to

+ * be the port number (usually 5672) on which the RabbitMQ endpoint can be reached.

+ */

+ String RABBITMQ_TAG_NAME = "RabbitMQEndpoint";

+

+ <h2 class="articleSubheadline">January 2024</h2>

+ <ul class="bulletList">

+ <li>When launching a new application replica set in a region, the choice of the default RabbitMQ

+ server now depends on the region: in our "default" region "eu-west-1", RabbitMQ is identified

+ by the DNS-mapped host name "rabbit.internal.sapsailing.com". Elsewhere, the RabbitMQ server

+ in the region is explored using the RabbitMQEndpoint tag. If not found, again

+ "rabbit.internal.sapsailing.com" will be used, assuming there may be a VPC peering across

+ regions.</li>

+ </ul>

+import com.sap.sse.landscape.rabbitmq.RabbitMQEndpoint;

+

+ @Test

+ public void getDefaultRabbitConfigForEuWest1() {

+ final RabbitMQEndpoint rabbitConfig = landscape.getDefaultRabbitConfiguration(new AwsRegion(Region.EU_WEST_1, landscape));

+ assertEquals("rabbit.internal.sapsailing.com", rabbitConfig.getNodeName());

+ assertEquals(5672, rabbitConfig.getPort());

+ }

+

+ @Test

+ public void getDefaultRabbitConfigForEuWest2() {

+ final RabbitMQEndpoint rabbitConfig = landscape.getDefaultRabbitConfiguration(new AwsRegion(Region.EU_WEST_2, landscape));

+ assertTrue(rabbitConfig.getNodeName().startsWith("172.31."));

+ assertEquals(5672, rabbitConfig.getPort());

+ }

+

+ @Test

+ public void getDefaultRabbitConfigForRegionWithNoTaggedInstanceInIt() {

+ final RabbitMQEndpoint rabbitConfig = landscape.getDefaultRabbitConfiguration(new AwsRegion(Region.US_EAST_2, landscape));

+ assertEquals("rabbit.internal.sapsailing.com", rabbitConfig.getNodeName());

+ assertEquals(5672, rabbitConfig.getPort());

+ }

+ org.mongodb.driver-sync;bundle-version="4.3.1",

+ com.sap.sailing.landscape.common

+import com.sap.sailing.landscape.common.SharedLandscapeConstants;

+ public RabbitMQEndpoint getDefaultRabbitConfiguration(com.sap.sse.landscape.Region region) {

+ final RabbitMQEndpoint defaultRabbitMQInDefaultRegion = ()->SharedLandscapeConstants.RABBIT_IN_DEFAULT_REGION_HOSTNAME; // using default port RabbitMQEndpoint.DEFAULT_PORT

+ if (region.getId().equals(Region.EU_WEST_1.id())) {

+ result = defaultRabbitMQInDefaultRegion;

+ final Iterable<AwsInstance<ShardingKey>> rabbitMQHostsInRegion = getRunningHostsWithTag(

+ region, SharedLandscapeConstants.RABBITMQ_TAG_NAME, AwsInstanceImpl::new);

+ if (rabbitMQHostsInRegion.iterator().hasNext()) {

+ final AwsInstance<ShardingKey> anyRabbitMQHost = rabbitMQHostsInRegion.iterator().next();

+ result = new RabbitMQEndpoint() {

+ @Override

+ public int getPort() {

+ return getTag(anyRabbitMQHost, SharedLandscapeConstants.RABBITMQ_TAG_NAME)

+ .map(t -> t.trim().isEmpty() ? RabbitMQEndpoint.DEFAULT_PORT : Integer.valueOf(t.trim()))

+ .orElse(RabbitMQEndpoint.DEFAULT_PORT);

+ }

+

+ @Override

+ public String getNodeName() {

+ return anyRabbitMQHost.getPrivateAddress().getHostAddress();

+ }

+ };

+ } else {

+ result = defaultRabbitMQInDefaultRegion; // no instance with tag found; hope for VPC peering and use RabbitMQ hostname from default region

+ }

+import com.sap.sse.landscape.Region;

+ * {@link AwsLandscape#getDefaultRabbitConfiguration(Region) default RabbitMQ

+ * message channels and exchanges.<p>

+ *

+ * For our default region, this will return a DNS name always pointing to the current private IP of

+ * the instance running the default RabbitMQ service in the region. In other regions, the private IP

+ * of the regional default RabbitMQ instance is discovered by scanning for running instances tagged

+ * with {@link SharedLandscapeConstants#RABBITMQ_TAG_NAME}.

+ RabbitMQEndpoint getDefaultRabbitConfiguration(Region region);

+

+- SAP Sailing Analytics, ``image-type`` is ``sailing-analytics-server``, see [here](/wiki/info/landscape/creating-ec2-image-from-scratch)

+- MongoDB Live Replica Set NVMe, ``image-type`` is ``mongodb-server``, see [here](/wiki/info/landscape/creating-ec2-mongodb-image-from-scratch)

+- Hudson Debian/Ubuntu Slave, ``image-type`` is ``hudson-slave``

+- Webserver, ``image-type`` is ``webserver``, see [here](/wiki/info/landscape/creating-ec2-image-for-webserver-from-scratch)

+

+There are furthermore instance types that we can configure automatically, based on a clean Amazon Linux 2 instance launched from the respective default Amazon image:

+- Hudson / dev.sapsailing.com server, see [here](/wiki/info/landscape/creating-ec2-image-for-hudson-from-scratch)

+- MySQL / MariaDB database server holding the data for our ``bugzilla.sapsailing.com`` bug/issue tracker, see [here](/wiki/info/landscape/creating-ec2-image-for-mysql-from-scratch)

+- RabbitMQ default instance used by all default sailing servers for replication, see [here](/wiki/info/landscape/creating-ec2-image-for-rabbitmq-from-scratch)

+

+We try to maintain setup scripts that help us with setting up those instance types from scratch. See the respective Wiki pages referenced from the lists above for more details.

+# Setting up an Instance for the MySQL / MariaDB Bugzilla Database

+Our Bugzilla system at [bugzilla.sapsailing.com](https://bugzilla.sapsailing.com) uses a relational database to store all the bugs and issues. This used to be a MySQL database and has been migrated to MariaDB at the beginning of 2024.

+We don't provide a dedicated AMI for this because we don't need to scale this out or replicate this by any means. Instead, we provide a script to set this up, starting from a clean Amazon Linux 2 instance.

+

+Launch a new instance, based on the latest Amazon Linux 2 AMI maintained by AWS, and configure the root volume size to be, e.g., 16GB. As of this writing, the total size consumed by the database contents on disk is less than 1GB. Tag the volume with a tag key ``WeeklySailingInfrastructureBackup`` and value ``Yes`` to include it in the weekly backup schedule.

+

+When the instance has finished booting up, run the following script, passing the external IP address of the instance as mandatory argument:

+```

+ configuration/mysql_instance_setup/setup-mysql-server.sh a.b.c.d

+```

+where ``a.b.c.d`` stands for the external IP address you have to specify. Before the IP address you may optionally specify the passwords for the ``root`` and the ``bugs`` user of the existing database to be cloned to the new instance. Provide the ``root`` password with the ``-r`` option, the ``bugs`` password with the ``-b`` option. Passwords not provided this way will be prompted for.

+

+The script will then transfer itself to the instance and execute itself there, forwarding the passwords required. On the instance, it will then establish the periodic management of the login user's ``authorized_keys`` file for all landscape managers' keys, install the packages required (in particular mariadb105-server and cronie), then run a backup on the existing ``mysql.internal.sapsailing.com`` database using the ``root`` user and its password. The ``mysqldump`` client for this is run on ``sapsailing.com``, and the result is stored in the ``/tmp`` folder on the new instance where it is then imported. The import is a bit tricky in case this is a migration from MySQL to MariaDB where the users table has become a view. Therefore, a few additional ``DROP TABLE`` and ``DROP VIEW`` commands are issued before importing the data. When the import is complete, user privileges are flushed so they match with what has been imported. The DB is then re-started in "safe" mode so that the user passwords can be adjusted, in case this was a migration from MySQL to MariaDB. Finally, the DB is restarted properly with the new user passwords.

+

+The instance then is generally available for testing. Run a few ``mysql`` commands, check out the ``bugs`` database and its contents, especially those of the ``bugs.bugs`` table. If this all looks good, switch the DNS record for ``mysql.internal.sapsailing.com`` to the private IP of the new instance. This will be used by the Bugzilla installation running on our central reverse proxy. When this is done you can consider stopping and ultimately terminating the old DB server.

+# Setting up a RabbitMQ Server Instance

+RabbitMQ is hard to install on latest versions of Amazon Linux (e.g., 2, or 2023). Therefore, we use a latest Debian 12 default image to start with.

+Configure the root volume to be at least 8GB. The empty installation takes about 1.6GB, so you will have enough room for messages queued persistently.

+

+When the instance has finished booting and SSH access is possible, invoke the following script, providing the instance's external IP address as only parameter:

+```

+ configuration/rabbitmq_instance_setup/setup-rabbitmq-server.sh a.b.c.d``

+```

+where ``a.b.c.d`` is the external IP address of your fresh instance.

+

+The script will ensure the login user's ``authorized_keys`` are updated periodically to contain those of the landscape managers, then will install the necessary packages, particularly ``rabbitmq-server`` and, to get real log files under ``/var/log``, the ``syslog-ng`` package. It then enables the ``rabbitmq_management`` plugin, so access to the management UI becomes possible through port ``15672``. The configuration file under ``/etc/rabbitmq/rabbitmq.conf`` is patched such that guest logins are possible also from non-localhost addresses, by adding the ``loopback_users = none`` directive to the config file. It finally (re-)starts the RabbitMQ server to let these config changes take effect.

+

+Your RabbitMQ server then should be ready to handle requests. Test this by invoking the management UI, e.g., through an ssh port forward to port ``15672``. When this seems good, pick a smart time to change the DNS record for ``rabbit.internal.sapsailing.com`` because there will be a short time of interruptions on all application processes currently connected to the old RabbitMQ which you then have to stop. Those client applications will temporarily lose connection, but our replication component will re-establish these connections, using the DNS name which gets resolved again based on the DNS entry's TTL.

+

+Then associate the elastic IP ``54.76.64.42`` as the external IP of the new instance. This will let ``rabbit.sapsailing.com`` point to the public IP of the instance.

+

+Add a tag with key ``RabbitMQEndpoint`` and value ``5672``, specifying the port on which the RabbitMQ server listens. This tag can be used by our landscape automation procedures to discover the RabbitMQ default instance in the region.