completed Wiki description of MongoDB major version upgrade

+Upgrading a MongoDB replica set that has more than one node can work without client noticing any interruption of service. This is in particular important for our ``live`` replica set used by all running application replica sets other than ``ARCHIVE``. For the single-node replica sets ``archive`` and ``slow`` it again comes down to timing an upgrade such that no ``ARCHIVE`` candidate launch is ongoing, and that no application replica set is currently being shut down with its database getting backed up to the ``slow`` replica set.

+

+The [MongoDB online documentation](https://www.mongodb.com/docs/manual/release-notes/8.0-upgrade-replica-set/#std-label-8.0-upgrade-replica-set) contains a useful description of the steps necessary. The key to understanding those steps is that MongoDB replica sets can distinguish between the actual version of ``mongod`` that is running, and the "protocol version" the nodes use to talk to each other in a replica set. Newer ``mongod`` versions can always still work with the "protocol version" of the previous major release. For example, ``mongod`` in version 8 can still work with protocol version "7.0".

+

+Therefore, upgrading a replica set with multiple nodes will work along these steps:

+

+- Ensure all ``mongod`` processes in the replica set run the same (old) version

+- Ensure all ``mongod`` processes use the protocol version that matches their own ``mongod`` version

+- Upgrade the binaries and restart the ``mongod`` processes with the new version for all nodes, properly having the primary step down before restarting its process

+- Set the new protocol version for the replica set

+

+The sequence in which to work with the different nodes and processes resembles that for reboots after upgrades with the package manager. Here are the steps in detail:

+

+- Ensure that no ARCHIVE candidate is currently launching; such a candidate would read from the ``archive`` replica set, so that rebooting the ``dbserver.internal.sapsailing.com`` node would interrupt this loading process. If an ARCHIVE candidate is launching, wait for the launch to finish.

+- Ensure that no application replica set is currently being shut down with backing up its database. This backup would fail if the ``dbserver.internal.sapsailing.com`` node were restarted as it hosts the ``slow`` replica set used for the backup.

+- ssh into ``ec2-user@dbserver.internal.sapsailing.com``

+- Ensure all ``mongod`` processes on the host run the same (old) version, using ``mongosh`` for all three replica sets (``live``, ``archive``, ``slow``)

+- In ``mongosh``, display the protocol version using ``db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )``. Should you find a deviation, set the protocol version using ``db.adminCommand( { setFeatureCompatibilityVersion: "7.0" , confirm: true } )`` (of course with the "7.0" replaced by whichever protocol version you have to set this to).

+- In ``/etc/yum.repos.d/`` find the ``mongodb-org.{major.minor}.repo`` file that controls where the MongoDB packages are currently obtained from. Rename the current ``mongodb-org.{major.minor}.repo`` by appending, e.g., ``.bak`` to its name and create a new ``.repo`` file for the MongoDB version you'd like to upgrade to. Then run ``dnf --releasever=latest upgrade``. This should automatically restart the ``mongod`` processes now upgraded to the new release.

+- ssh into ``ec2-user@mongo0.internal.sapsailing.com``

+- check ``mongod`` and protocol version using ``mongosh``; adjust protocol version if necessary (see above)

+- if on the "primary", use ``rs.stepDown()`` to make it a "secondary"

+- run the binaries upgrade as explained for ``dbserver.internal.sapsailing.com`` above, adjusting the ``.repo`` file under ``/etc/yum.repos.d``, followed by ``dnf --releasever=latest upgrade``

+- repeat the last four steps for ``mongo1.internal.sapsailing.com``

+- use ``mongosh`` to connect to the primaries of all three replica sets (``live``, ``archive``, ``slow``) and on each one issue the command ``db.adminCommand( { setFeatureCompatibilityVersion: "8.0", confirm: true }`` with the "8.0" replaced by the protocol version you want to upgrade to, so usually the major/minor version of the binaries to which you have upgraded.

+

+Done :-)