Setting up an EC2 Image for a MongoDB Replica Set from Scratch

Image and Basic Packages

Start with an Amazon Linux 2 AMI (HVM), SSD Volume Type - ami-0fad7378adf284ce0 (64-bit x86) image.

Add the yum repository for MongoDB 3.6 by creating /etc/yum.repos.d/mongodb-org-3.6.repo as follow:

[mongodb-org-3.6]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.6/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc

Then:

yum install -y mongodb-org-server mongodb-org-mongos mongodb-org-shell mongodb-org-tools
mkfs.xfs /dev/nvme0n1
mount /dev/nvme0n1 /var/lib/mongo

MongoDB Set-Up

In file /etc/mongod.conf comment the line 

#  bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.
and instead add 
bindIp: 0.0.0.0

so that MongoDB listens on all interfaces, not only localhost. Furthermore, set the directoryPerDB property to true and provide a replication set name using the replication.replSetName property. The MongoDB default would be rs0. In our landscape we're using archive for the winddb database on dbserver.internal.sapsailing.com:10201 and live for all live events with a hidden replica at dbserver.internal.sapsailing.com:10203.

Here is the full /etc/mongod.conf file:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
  directoryPerDB: true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 27017
#  bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.
  bindIp: 0.0.0.0

#security:

#operationProfiling:

replication:
  replSetName: live

#sharding:

## Enterprise-Only Options

#auditLog:

#snmp:

To deal with the set-up of the ephemeral volume, there are a number of systemd units that can be found in the git repository at configuration/mongo_instance_setup. The ephemeralvolume.service ensures that the volume exists and is formatted with the XFS file system. It looks for the volume at /dev/nvme0n1 and on /dev/xvdb, ensures XFS formatting and then mounts the partition to /var/lib/mongo. This service uses the /usr/local/bin/ephemeralvolume script to do so. The ownerships are adjusted after that by the chownvarlibmongo.service unit to the mongod user. This is then also the last unit required to run before the mongod.service can start. After that, the replica set hook-up needs to take place. For that, the unit mongo-replica-set.service is provided, together with the two scripts /usr/local/bin/add-as-replica and /usr/local/bin/remove-as-replica. The service unit is configured so that it is "wanted" by multi-user.target, making

Then execute systemctl enable mongod.service to launch the MongoDB process and ensure it is always launched when the instance (re-)boots.

MongoDB Replica Set Configuration

Connect to the MongoDB on that instance, then issue the command rs.initiate() in order to turn the instance into the "seed" of a replica set. You can then, for the time being, quit() the mongo shell. Re-connecting, e.g., with mongo "mongodb://localhost:27017/?replicaSet=live", will show the PRIMARY of the new replica set.

Connected to the PRIMARY using the mongo shell, a replica can be added using rs.add({host: "hostname:port"}). For additional options see here. To add a hidden replica on dbserver.internal.sapsailing.com:10203, use rs.add({host: "dbserver.internal.sapsailing.com:10203", buildIndexes: false, hidden: true, priority: 0}).

TODO: automate the initialization and replica set extension using "Addition Details" in the instance; create a MongoDB script that is executed during start-up; if no replica set exists and no user detail specifies where the primary is, run rs.initiate(). If a replica set already exists, leave things unchanged. If no replica set exists and in a user detail something like REPLICA_SET_NAME=... and REPLICA_SET_PRIMARY=... is provided, add the local node as a secondary to the primary / replica set specified.

Launch Script for a MongoDB Replica

55min initial replication for archive server winddb with dbserver.internal.sapsailing.com:10201/winddb using a gp2 SSD; additional 20min to build index