ec2-docs: interims changes, committed and pushed only for backup

+Our default region in AWS EC2 is eu-west-1 (Ireland). Tests are currently run in the otherwise unused region eu-west-2 (London). Most regular operations can be handled through the AdminConsole's "Advanced / Landscape" tab. See, e.g., [https://security-service.sapsailing.com/gwt/AdminConsole.html#LandscapeManagementPlace:](https://security-service.sapsailing.com/gwt/AdminConsole.html#LandscapeManagementPlace:). Some operations occurring not so frequently still require more in-depth knowledge of steps, manual execution of commands on the command line and some basic Linux understanding.

+## Important Servers, Hostnames

+- MySQL DB (mainly for Bugzilla): mysql.internal.sapsailing.com (currently co-deployed on the same old instance that also runs RabbitMQ, hence currently mysql.internal.sapsailing.com and rabbit.internal.sapsailing.com refer to the same instance)

+- Hudson Build Server: called "Build/Test/Dev", running a Hudson instance reachable at ``hudson.sapsailing.com`` and a test instance of the SAP Sailing Analytics available under ``dev.sapsailing.com``

+- Additional "Build Slaves" launched by the Hudson Build Server: Named ``Hudson Ubuntu Slave``, used to run individual build jobs

+Additionally, we have created a CNAME record for ``*.sapsailing.com`` pointing at a default application load balancer (ALB) (currently ``DefDynsapsailing-com-1492504005.eu-west-1.elb.amazonaws.com``) in our default region (eu-west-1). Thie default ALB is also called our "dynamic ALB" because it doesn't depend on DNS rules other than the default one for ``*.sapsailing.com``, so other than changes to the DNS which can take minutes to hours to propagate through the world-wide DNS, changes to the default ALB's rule set take effect immediately. Like all ALBs, this one also has a default rule that refers all traffic not matched by other rules to a target group that forwards traffic to an (in the future probably multiple) Apache httpd webserver. All these ALBs handle SSL termination by means of an ACM-managed certificate that AWS automatically renews before it expires. The traffic routed to the target groups is always HTTP only.

+Further ALBs may exist in addition to the default ALB and the NLB for ``sapsailing.com``. Those will then have to have one or more DNS record(s) pointing to them for which matching rules based on the hostname exist in the ALB listener's rule set. This set-up is specifically appropriate for "longer-lived" content where during archiving or dismantling a DNS lag is not a significant problem.

+

+### Apache httpd Webserver and Reverse Proxy

+

+The web server currently exists only as one instance but could now be replicated to other availability zones (AZ)s, entering those other IPs into the ``HTTP-to-sapsailing-dot-com`` target group (and, as will be described further below, to the ``CentralWebServerHTTP*`` (for the "dynamic" ALB in eu-west-1) or ``{ALB-name}-HTTP`` (for all DNS-mapped ALBs) target group of each application load balancer (ALB) in the region). For all of sapsailing.com it does not (no longer) care about SSL and does not need to have an SSL certificate (anymore). In particular, it offers the following services:

+Furthermore, it hosts aliases for ``sapsailing.com``, ``www.sapsailing.com`` and all subdomains for archived content, pointing to the archive server which is defined in ``/etc/httpd/conf.d/000-macros.conf``. This is also where the archive server switching has to be configured. Before reloading the configuration, make sure the syntax is correct, or else you may end up killing the web server, leading to downtime. Check by running

+* as regular instance target in all load balancers' default rule's target group, such as ``DefDynsapsailing-com``, ``DNSMapped-0``, ``DNSMapped-1``, and so on; the names of the target groups are ``CentralWebServerHTTP-Dyn``, ``DDNSMapped-0-HTTP``, ``DDNSMapped-1-HTTP``, and so on, respectively.

+We distinguish between DNS-mapped and non-DNS-mapped content. The basic services offered by the web server as listed above are DNS-mapped, with the DNS entries being CNAME records pointing to an ALB (DNSMapped-0-1286577811.eu-west-1.elb.amazonaws.com) which handles SSL offloading with the Amazon-managed certificate and forwards those requests to the web server. Furthermore, longer-running application replica sets can have a sub-domain declared in Route53's DNS, pointing to an ALB which then forwards to the public and master target groups for this replica set based on hostname, header fields and request method. A default redirect for the ``/`` path can also be defined, obsoleting previous Apache httpd reverse proxy redirects for non-archived ALB-mapped content.

+The requests going straight to ``sapsailing.com`` are handled by the NLB (see above), get forwarded to the web server and are re-directed to ``www.sapsailing.com`` from there, ending up at the non-DNS-mapped load balancer where by default they are then sent again to the web server / reverse proxy which sends it to the archive server.

+

+In addition to a default re-direct for the "/" path, the following four ALB listener rules for a single application replica set are defined, all requiring the "Host" to match the hostname:

+- if the HTTP header ``X-SAPSSE-Forward-Request-To`` is ``master`` then forward to the master target group

+- if the HTTP header ``X-SAPSSE-Forward-Request-To`` is ``replica`` then forward to the public target group

+- if the request method is ``GET`` then forward to the public target group

+- forward all other request for the hostname to the master target group

+

+### MongoDB Replica Sets

+

+### Shared Security and Application Data Across ``sapsailing.com``

+

+TODO explain the special role of security-service.sapsailing.com

+

+### Dedicated Application Replica Set

+

+### Application Replica Set Using Shared Instances

+

+### Auto-Scaling Groups and Launch Configurations

+

+### Important Amazon Machine Images (AMIs)

+

+TODO talk about image upgradability using the "image-upgrade" user data line, optionally combined with the "no-shutdown" flag.

+

+### AWS Tags

+

+AMIs / image-type and versions

+

+sailing-analytics-server

+

+mongo-replica-sets

+

+## Automated Procedures

+

+### Creating a New Application Replica Set

+

+### Moving Application Replica Set from Shared to Dedicated Infrastructure

+

+### Moving Application Replica Set from Dedicated to Shared Infrastructure

+

+### Scaling Replica Instances Up/Down

+

+### Scaling Master Up/Down

+

+### Upgrading Application Replica Set

+

+### Archiving Application Replica Set

+

+### Removing Application Replica Set

+

+### Upgrading Application AMI

+

+TODO explain how launch configurations can be upgraded as well

+

+### Upgrading MongoDB AMI

+## Current Status

+

+Parts of the plan outlined by this document around August 2018 have been implemented by February 2022. The document is still left in place because it contains several concepts and ideas not implemented yet. The following things have already been achieved by February 2022:

+

+- SSL offloading at the ALB: all application traffic is taken in through application load balancers (ALBs) by now. The HTTPS/SSL connection is terminated there, using an AWS-managed certificate with automatic renewal.

+- An Apache-based central reverse proxy exists only for archived content and a few basic services such as CI, Bug tracking or HTTPS Git access

+- Automation functionality has been built using the Java/OSGi-based architecture suggested herein. It has been integrated in the existing AdminConsole for now, using a new "Landscape" tab in the "Advanced" category.

+- Automation covers handling and upgrading AMIs (images), DNS records, ALBs, creating and managing shared ("multi") instances, creating and managing auto-scaling groups, migrating application replica sets from shared to dedicated infrastructure and back, scaling master and replicas up and down, archiving content from a replica set into an archive server, scaling MongoDB replica sets up and down, and upgrading replica sets to new application versions.

+- Logging happens primarily from ALB to S3; a synchronization script under ``/var/log/old/cache/aws/sync-alb-access-logs-from-s3.sh`` which is also found in the git folder ``configuration/`` handles the synchronization

+-

+

+Major topics yet missing:

+

+- Making all relevant AMIs automatically upgradable (Webserver, Hudson build server, MongoDB)

+- Archive server handling (in particular upgrades and automatic failover)

+- Routing based on criteria other than hostname (e.g., in order to allow for more dynamic "scope migrations")

+- The central reverse proxy is still a single point of failure and should be replicated, based on a shared configuration

+- Cross-region set-ups will still require manual preparation (getting at least the sailing server AMI there, setting up security group, avoiding DNS interference if a Global Accelerator is being used, not setting up a master node for an application replica set in every region but only in a primary region or even on site, setting up the auto-scaling group and managing it during upgrades, ...); for the Tokyo 2020 Olympic Games we used scripts found in Git under ``configuration/on-site-scripts`` that helped during the process.

+- MongoDB disk space management automation or at least support

+- MongoDB replica set upgrades

+- automation of AMI production "from scratch" for reverse proxy nodes, MongoDB hosts and application servers

+- automatic management of alarms for every target group created

+