improved Wiki description of MongoDB setup and replica launch script Change-Id: Ia8eedbfae1a3da40b2e644805d698694e2d5121f

+# launch-mongodb-replica.sh [ <replica-set-name> [ <primary-host:primary-port> [ <priority> ] ] ]

+# The defaults are: live, mongo0.internal.sapsailing.com:27017, 1

+#

+# Examples:

+# launch-mongodb-replica.sh

+# connects to the "live" replica set through mongo0.internal.sapsailing.com:27017 with priority 1

+# launch-mongodb-replica.sh archive dbserver.internal.sapsailing.com:10201 0

+# connects to the "archive" replica set at dbserver.internal.sapsailing.com:10201 and ensures

+# that the new replica never becomes PRIMARY by setting the priority to 0

+REPLICA_SET_PRIMARY=$2

+REPLICA_SET_PRIORITY=$3" --ebs-optimized --key-name Axel

+ REPLICA_SET_PRIORITY=1

+To deal with the set-up of the ephemeral volume, there are a number of ``systemd`` units that can be found in the git repository at ``configuration/mongo_instance_setup``. Copy the ``*.service`` files to ``/etc/systemd/system`` and the remaining scripts to ``/usr/local/bin``.

+The ``ephemeralvolume.service`` ensures that the volume exists and is formatted with the XFS file system. It looks for the volume at ``/dev/nvme0n1`` and on ``/dev/xvdb``, ensures XFS formatting and then mounts the partition to ``/var/lib/mongo``. This service uses the ``/usr/local/bin/ephemeralvolume`` script to do so. The ownerships are adjusted after that by the ``chownvarlibmongo.service`` unit to the ``mongod`` user. This is then also the last unit required to run before the ``mongod.service`` can start. After that, the replica set hook-up needs to take place. For that, the unit ``mongo-replica-set.service`` is provided, together with the two scripts ``/usr/local/bin/add-as-replica`` and ``/usr/local/bin/remove-as-replica``. The service unit is configured so that it is "wanted" by ``multi-user.target``, making it start automatically when the system comes up. It depends on ``mongod.service``, hence waits until the MongoDB process has started successfully. It then registers as a replica in a replica set. The following user data variables can be used to configure the behavior:

+- ``REPLICA_SET_NAME``: the replica set name to which to subscribe; this name will also be patched into the ``/etc/mongod.conf`` config file; defaults to ``live``

+- ``REPLICA_SET_PRIMARY``: the hostname and port, separated by a colon ``:``, of the replica set's primary to which to connect; defaults to ``mongo0.internal.sapsailing.com:27017``

+- ``REPLICA_SET_PRIORITY``: the priority with which the new replica shall become a master; defaults to ``1`` which lets the new replica become master. Set it to ``0`` to avoid ever promoting the new replica to a primary.

+Then execute ``systemctl enable mongod.service`` and ``systemctl enable mongo-replica-set.service`` to launch the MongoDB process and ensure it is always launched when the instance (re-)boots and becomes a replica in a replica set according to the user data and the defaults described above.

+## Manual MongoDB Replica Set Configuration

+In case you decide to skip the benefits of ``mongo-replica-set.service``, you can also establish the connection with the replica set as follows. Connect to the MongoDB on the instance. If it is the first instance of a replica set, issue the command ``rs.initiate()`` in order to turn the instance into the "seed" of a replica set. You can then, for the time being, ``quit()`` the mongo shell. Re-connecting, e.g., with ``mongo "mongodb://localhost:27017/?replicaSet=live"``, will show the ``PRIMARY`` of the new replica set. Don't issue these commands if you only want to add another replica to an existing replica set.

+Connected to the PRIMARY using the ``mongo`` shell, a replica can be added using ``rs.add({host: "hostname:port"})``. For additional options see [here](https://docs.mongodb.com/manual/reference/method/rs.add/). To add a hidden replica on ``dbserver.internal.sapsailing.com:10203``, use ``rs.add({host: "dbserver.internal.sapsailing.com:10203", buildIndexes: false, hidden: true, priority: 0})``.

+Under ``configuration/aws-automation/launch-mongodb-replica.sh`` there is a script that can be used for quickly launching a MongoDB replica. Usage:```

+ launch-mongodb-replica.sh [ &lt;replica-set-name&gt; [ &lt;primary-host:primary-port&gt; [ &lt;priority&gt; ] ] ]

+```

+The defaults for the parameters are, respectively: ``live``, ``mongo0.internal.sapsailing.com:27017``, and ``1``.

+

+This can, e.g., be used to fire up a replica for the ``winddb`` instance on ``dbserver.internal.sapsailing.com:10201`` which is configured to run on a slow, inexpensive but large disk and on a host that has only little RAM configured because the only load it has to serve may be occasional master data import activity and burst-loads during archive server re-starts. To handle the latter, an instance with more RAM and fast disk would be preferable because otherwise indices won't fit into RAM, letting MongoDB read indices from the slow disk. This lets the archive server take many days to load.

+Instead, the ``launch-mongodb-replica.sh`` script can be used to fire up a replica with a very fast NMVe SSD attached to the instance. A good instance type for this may be ``i2.xlarge`` which has 30GB of RAM, 4 vCPUs and 800GB of NVMe storage. Launch like this:

+``configuration/aws-automation/launch-mongodb-replica.sh archive dbserver.internal.sapsailing.com:10201 0``

+You can then check the replication status by connecting your ``mongo`` client to ``dbserver.internal.sapsailing.com:10201`` and check the output of ``rs.status()``. As long as the new instance is listed in status ``STARTUP2`` it is still synchronizing from the primary. Eventually (took 55min during the last test) it transitions to status ``SECONDARY``. You can then use a connection string such as ``mongodb://dbserver.internal.sapsailing.com:10201/winddb?replicaSet=archive&retryWrites=true&readPreference=secondary`` for the ``MONGODB_URI`` in your archive server Java instance which will then preferably connect to your new fast secondary replica. Consider that the new secondary replica will have to build indices when first queried. This can take another 20 minutes. When done loading the archive server, terminate the instance which will properly remove the replica from the replica set.