wiki/info/landscape/creating-ec2-image-for-hudson-from-scratch.md
... ...
@@ -0,0 +1,112 @@
1
+# Setting up an image for the hudson.sapsailing.com server
2
+
3
+This is an add-on to the regular EC2 image set-up described [here](https://wiki.sapsailing.com/wiki/info/landscape/creating-ec2-image-from-scratch). An Android SDK needs to be installed.
4
+
5
+
6
+* Create a ``hudson`` user/group
7
+* Make sure ``/home/hudson`` is a separate mount; probably just mount the existing volume of a previous installation
8
+* Install an Android SDK under ``/home/hudson/android-sdk-linux``; if you simply re-used an old ``/home/hudson`` mount this should already be in place.
9
+* Install Eclipse to ``/home/hudson/eclipse`` to allow sharing it in case a large AWS instance is needed, e.g., for heap dump analysis.
10
+* export ``/home/hudson/android-sdk-linux`` and ``/home/hudson/eclipse`` as follows in ``/etc/exports``:
11
+```
12
+/home/hudson/android-sdk-linux 172.31.0.0/16(rw,nohide,no_root_squash)
13
+/home/hudson/eclipse 172.31.0.0/16(rw,nohide,no_root_squash)
14
+```
15
+
16
+* Ensure you have EC2 / EBS snapshot backups for the volumes by tagging them as follows: ``WeeklySailingInfrastructureBackup=Yes`` for ``/`` and ``/home/hudson``.
17
+
18
+``/home/hudson/repo`` has the Hudson build repository. The Hudson WAR file is under ``/usr/lib/hudson/hudson.war``. ``/etc/init.d/hudson``, linked to from ``/etc/rc0.d/K29hudson``, ``/etc/rc1.d/K29hudson``, ``/etc/rc2.d/K29hudson``, ``/etc/rc3.d/S81hudson``, ``/etc/rc4.d/K29hudson``, ``/etc/rc5.d/S81hudson``, and ``/etc/rc6.d/K29hudson``, takes care of spinning up Hudson during instance re-boot. Hudson systemwide configuration is under ``/etc/sysconfig/hudson``:
19
+```
20
+## Path: Development/Hudson
21
+## Description: Configuration for the Hudson continuous build server
22
+## Type: string
23
+## Default: "/var/lib/hudson"
24
+## ServiceRestart: hudson
25
+#
26
+# Directory where Hudson store its configuration and working
27
+# files (checkouts, build reports, artifacts, ...).
28
+#
29
+HUDSON_HOME="/home/hudson/repo"
30
+
31
+## Type: string
32
+## Default: ""
33
+## ServiceRestart: hudson
34
+#
35
+# Java executable to run Hudson
36
+# When left empty, we'll try to find the suitable Java.
37
+#
38
+
39
+HUDSON_JAVA_CMD="/opt/sapjvm_8/bin/java"
40
+# The following line choses JavaSE-1.7
41
+#HUDSON_JAVA_CMD="/opt/jdk1.7.0_02/bin/java"
42
+# The following line choses JavaSE-1.8
43
+#HUDSON_JAVA_CMD="/opt/jdk1.8.0_20/bin/java"
44
+
45
+## Type: string
46
+## Default: "hudson"
47
+## ServiceRestart: hudson
48
+#
49
+# Unix user account that runs the Hudson daemon
50
+# Be careful when you change this, as you need to update
51
+# permissions of $HUDSON_HOME and /var/log/hudson.
52
+#
53
+HUDSON_USER="hudson"
54
+
55
+## Type: string
56
+## Default: "-Djava.awt.headless=true"
57
+## ServiceRestart: hudson
58
+#
59
+# Options to pass to java when running Hudson.
60
+#
61
+HUDSON_JAVA_OPTIONS="-Djava.awt.headless=true -Xmx2G -Dhudson.slaves.ChannelPinger.pingInterval=60 -Dhudson.slaves.ChannelPinger.pingIntervalSeconds=60 -Dhudson.slaves.ChannelPinger.pingTimeoutSeconds=60"
62
+
63
+## Type: integer(0:65535)
64
+## Default: 8080
65
+## ServiceRestart: hudson
66
+#
67
+# Port Hudson is listening on.
68
+#
69
+HUDSON_PORT="8080"
70
+
71
+## Type: integer(1:9)
72
+## Default: 5
73
+## ServiceRestart: hudson
74
+#
75
+# Debug level for logs -- the higher the value, the more verbose.
76
+# 5 is INFO.
77
+#
78
+HUDSON_DEBUG_LEVEL="5"
79
+
80
+## Type: yesno
81
+## Default: no
82
+## ServiceRestart: hudson
83
+#
84
+# Whether to enable access logging or not.
85
+#
86
+HUDSON_ENABLE_ACCESS_LOG="no"
87
+
88
+## Type: integer
89
+## Default: 100
90
+## ServiceRestart: hudson
91
+#
92
+# Maximum number of HTTP worker threads.
93
+#
94
+HUDSON_HANDLER_MAX="100"
95
+
96
+## Type: integer
97
+## Default: 20
98
+## ServiceRestart: hudson
99
+#
100
+# Maximum number of idle HTTP worker threads.
101
+#
102
+HUDSON_HANDLER_IDLE="20"
103
+
104
+## Type: string
105
+## Default: ""
106
+## ServiceRestart: hudson
107
+#
108
+# Pass arbitrary arguments to Hudson.
109
+# Full option list: java -jar hudson.war --help
110
+#
111
+HUDSON_ARGS=""
112
+```
wiki/info/landscape/creating-ec2-image-for-webserver-from-scratch.md
... ...
@@ -10,7 +10,13 @@ This is an add-on to the regular EC2 image set-up described [here](https://wiki.
10 10
11 11
Then carry out these steps:
12 12
13
-* install additional packages: `yum install fail2ban git mod24_perl perl perl-CGI perl-Template-Toolkit perl-HTML-Template perl-CPAN perl-DBD-MySQL mod24_ssl php71 php71-mysqlnd mod24-ldap ruby24 ruby24-devel rubygems24 rubygems24-devel icu libicu-devel gcc-c++ ncurses-devel geoip-devel perl-autodie`
13
+* install additional packages:
14
+```
15
+ yum install fail2ban git mod24_perl perl perl-CGI perl-Template-Toolkit perl-HTML-Template perl-CPAN perl-DBD-MySQL \
16
+ mod24_ssl php71 php71-mysqlnd mod24-ldap ruby24 ruby24-devel rubygems24 rubygems24-devel icu libicu-devel \
17
+ gcc-c++ ncurses-devel geoip-devel perl-autodie docker
18
+```
19
+
14 20
* activate NFS by calling `chkconfig nfs on`; ensure that `/var/log/old` and `/home/scores` are exposed in `/etc/exports` as follows:
15 21
```
16 22
/var/log/old 172.31.0.0/16(rw,nohide,no_root_squash)
... ...
@@ -64,10 +70,10 @@ sinatra-2.0.0 depends on rack (~> 2.0)
64 70
If you remove this gem, these dependencies will not be met.
65 71
Continue with Uninstall? [yN] y
66 72
Successfully uninstalled rack-2.0.3
67
-
68 73
```
69
-* ensure there are users and groups for `wiki`, `scores`, `wordpress`, `trac` that match up with their /home directory owners / groups
74
+* ensure there are users and groups for `wiki`, `scores`, `trac` that match up with their /home directory owners / groups
70 75
* ensure the Wiki startup script `serve.sh` configured for port 4567 and `config.ru` as well as the entire Gollum installation under /home/wiki are present, as well as the `users.yml` file
76
+* clone ``ssh://trac@sapsailing.com/home/trac/git`` into ``/home/wiki/gitwiki``
71 77
* ensure there is a reasonable `/root/.goaccess` file
72 78
* Configure goaccess by adjusting `/etc/goaccess.conf` such that it contains the following lines:
73 79
```
... ...
@@ -88,7 +94,49 @@ mv welcome.conf welcome.conf.org
88 94
```
89 95
* install bugzilla to `/usr/share/bugzilla` and `/var/lib/bugzilla`
90 96
* create `/etc/bugzilla/localconfig`
97
+* install scripts such as ``update_authorized_keys_for_landscape_managers_if_changed`` to ``/usr/local/bin``:
98
+```
99
+lrwxrwxrwx 1 root root 62 Jan 29 2022 awsmfalogon.sh -> /home/wiki/gitwiki/configuration/aws-automation/awsmfalogon.sh
100
+-r-xr-xr-x 1 root root 1465 Jan 11 2018 dbilogstrip
101
+-r-xr-xr-x 1 root root 6291 Jan 11 2018 dbiprof
102
+-r-xr-xr-x 1 root root 5479 Jan 11 2018 dbiproxy
103
+-rwxr-xr-x 1 root root 24707072 Jan 16 2022 docker-compose
104
+-r-xr-xr-x 1 root root 42043 Jan 11 2018 enc2xs
105
+-r-xr-xr-x 1 root root 3065 Jan 11 2018 encguess
106
+-rwxr-xr-x 1 root root 640 Jan 11 2018 github-markup
107
+-rwxr-xr-x 1 root root 598 Jan 11 2018 gollum
108
+-rwxr-xr-x 1 root root 613 Jan 11 2018 htmldiff
109
+-rwxr-xr-x 1 root root 610 Jan 11 2018 kramdown
110
+-rwxr-xr-x 1 root root 607 Jan 11 2018 ldiff
111
+-rwxr-xr-x 1 root root 352 Nov 1 2021 mail-events-on-my
112
+-rwxr-xr-x 1 root root 610 Jan 11 2018 mustache
113
+-rwxrwxr-x 1 trac trac 18992 Jun 16 2020 netio
114
+-rwxr-xr-x 1 root root 610 Jan 11 2018 nokogiri
115
+lrwxrwxrwx 1 root root 75 Oct 20 09:00 notify-operators -> /home/wiki/gitwiki/configuration/on-site-scripts/paris2024/notify-operators
116
+-r-xr-xr-x 1 root root 8356 Jan 11 2018 piconv
117
+-rwxr-xr-x 1 root root 648 Jan 11 2018 posix-spawn-benchmark
118
+-rwxr-xr-x 1 root root 590 Jan 11 2018 rackup
119
+-rwxr-xr-x 1 root root 596 Jan 11 2018 rougify
120
+-rwxr-xr-x 1 root root 616 Jan 11 2018 ruby-prof
121
+-rwxr-xr-x 1 root root 640 Jan 11 2018 ruby-prof-check-trace
122
+-rwxr-xr-x 1 root root 586 Jan 11 2018 tilt
123
+lrwxrwxrwx 1 root root 78 Feb 8 2021 update_authorized_keys_for_landscape_managers -> /home/wiki/gitwiki/configuration/update_authorized_keys_for_landscape_managers
124
+lrwxrwxrwx 1 root root 89 Feb 8 2021 update_authorized_keys_for_landscape_managers_if_changed -> /home/wiki/gitwiki/configuration/update_authorized_keys_for_landscape_managers_if_changed
125
+```
126
+* set up ``crontab`` for ``root`` user (remove the symbolic link to ``/home/sailing/code/configuration/crontab`` if that had been created earlier)
127
+```
128
+0 10 1 * * export PATH=/bin:/usr/bin:/usr/local/bin; mail-events-on-my >/dev/null 2>/dev/null
129
+* * * * * export PATH=/bin:/usr/bin:/usr/local/bin; sleep $(( $RANDOM * 60 / 32768 )); update_authorized_keys_for_landscape_managers_if_changed $( cat /root/ssh-key-reader.token ) https://security-service.sapsailing.com /root 2>&1 >>/var/log/sailing.err
130
+0 7 2 * * export PATH=/bin:/usr/bin:/usr/local/bin; docker exec -it registry-registry-1 registry garbage-collect /etc/docker/registry/config.yml
131
+```
91 132
* set up crontab for user `wiki` as `*/10 * * * * /home/wiki/syncgit` and make sure the script is in place
133
+* ensure that ``/var/log/old/cache/docker`` makes it across from any previous installation to the new one; it contains the docker registry contents. See in particular ``/var/log/old/cache/docker/registry/docker/registry/v2/repositories``.
134
+* [install docker registry](https://wiki.sapsailing.com/wiki/info/landscape/docker-registry) so that the following containers are up and running:
135
+```
136
+CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
137
+cd8086eb6361 joxit/docker-registry-ui:latest "/docker-entrypoint.…" 6 months ago Up 6 months 0.0.0.0:5000->80/tcp, :::5000->80/tcp registry-ui-1
138
+bcf1e278ecd7 registry:latest "/entrypoint.sh /etc…" 6 months ago Up 6 months 5000/tcp, 0.0.0.0:5001->5001/tcp, :::5001->5001/tcp registry-registry-1
139
+```
92 140
* ensure that `https://git.sapsailing.com/git` delivers the git content, with password credentials defined in `/etc/httpd/conf/passwd.git`. Sasa Zivkov (sasa.zivkov@sap.com) has been our point of contact of the SAP Gerrit group helping us with replicating our Git repository to the SAP-internal git.wdf.sap.corp one.
93 141
* comment `lbmethod_heartbeat_module` in /etc/httpd/conf.modules.d/00-proxy.conf because we don't need this sort of load balancing across origin servers and it causes a warning message in error_log
94 142
* install awstats to `/usr/share/awstats`, establish `/etc/httpd/conf/passwd.awstats`, establish a configuration under `/etc/awstats`, establish AWStats data directory under `/var/lib/awstats` and create /etc/cron.weekly/awstats as follows:
... ...
@@ -99,7 +147,6 @@ su -l -c '/usr/share/awstats/tools/awstats_updateall.pl now -configdir="
99 147
exit 0
100 148
```
101 149
* Follow the [mail setup](https://wiki.sapsailing.com/wiki/info/landscape/mail-relaying#setup-central-mail-server-instance-webserver) instructions
102
-* Install Wordpress
103 150
* Install gollum Wiki
104 151
* Copy git contents of ssh://trac@sapsailing.com/home/trac/git to /home/trac/git
105 152
* Ensure there is a /home/scores directory with subdirectories `barbados`, `kiwo`, `sailwave`, `scores`, `velum`, and `xrrftp`.
... ...
@@ -120,7 +167,4 @@ logpath = /var/log/secure
120 167
maxretry = 5
121 168
```
122 169
* Ensure that fail2ban will be started automatically when the instance starts: `chkconfig --level 23 fail2ban on` and start it right away with `service fail2ban start`. You can see which filters are active using `service fail2ban status`.
123
-
124
-## Appendix / Resources
125
-BACKUP_DIRECTORIES="/etc /home/trac/git /home/trac/mailinglists /home/trac/maven-repositories /home/trac/p2-repositories /home/trac/releases /home/trac/sapsailing_layouts.git /var/www/static /home/trac/crontab /home/scores /var/log/old"
126
-
170
+* Ensure you have EC2 / EBS snapshot backups for the volumes by tagging them as follows: ``WeeklySailingInfrastructureBackup=Yes`` for ``/var/www/static``, ``/var/log``, ``/var/log/old`` and ``/var/log/old/cache``, ``DailySailingBackup=Yes`` for ``/home``.
wiki/info/landscape/creating-ec2-image-from-scratch.md
... ...
@@ -2,6 +2,20 @@
2 2
3 3
I started out with a clean "Amazon Linux AMI 2015.03 (HVM), SSD Volume Type - ami-a10897d6" image from Amazon and added the existing Swap and Home snapshots as new volumes. The root/system volume I left as is, to start with. This requires having access to a user key that can be selected when launching the image.
4 4
5
+Add a ``sailing`` user / group. Under that user account, clone ``ssh://trac@sapsailing.com/home/trac/git`` to ``/home/sailing/code``.
6
+
7
+Under ``/usr/local/bin`` install the following:
8
+```
9
+lrwxrwxrwx 1 root root 56 Oct 20 09:20 cp_root_mail_properties -> /home/sailing/code/configuration/cp_root_mail_properties
10
+-rwxr-xr-x 1 root root 24707072 Jan 30 2022 docker-compose
11
+lrwxrwxrwx 1 root root 71 May 10 2021 getLatestImageOfType.sh -> /home/sailing/code/configuration/aws-automation/getLatestImageOfType.sh
12
+lrwxrwxrwx 1 root root 50 Mar 23 2021 launchhudsonslave -> /home/sailing/code/configuration/launchhudsonslave
13
+lrwxrwxrwx 1 root root 57 Mar 23 2021 launchhudsonslave-java11 -> /home/sailing/code/configuration/launchhudsonslave-java11
14
+lrwxrwxrwx 1 root root 69 Jun 1 2019 mountnvmeswap -> /home/sailing/code/configuration/archive_instance_setup/mountnvmeswap
15
+lrwxrwxrwx 1 root root 78 Jan 27 2021 update_authorized_keys_for_landscape_managers -> /home/sailing/code/configuration/update_authorized_keys_for_landscape_managers
16
+lrwxrwxrwx 1 root root 89 Feb 4 2021 update_authorized_keys_for_landscape_managers_if_changed -> /home/sailing/code/configuration/update_authorized_keys_for_landscape_managers_if_changed
17
+```
18
+
5 19
Enable the EPEL repository by issuing `yum-config-manager --enable epel/x86_64` or `sudo amazon-linux-extras install epel -y`.
6 20
7 21
I then did a `yum update` and added the following packages:
... ...
@@ -111,10 +125,10 @@ Added the following two lines to `/etc/security/limits.conf`:
111 125
112 126
```
113 127
* hard nproc unlimited
114
-* hard nofile 65000
128
+* hard nofile 128000
115 129
```
116 130
117
-This increases the maximum number of open files allowed from the default 1024 to a more appropriate 65k.
131
+This increases the maximum number of open files allowed from the default 1024 to a more appropriate 128k.
118 132
119 133
Copied the httpd configuration files `/etc/httpd/conf/httpd.conf`, `/etc/httpd/conf.d/000-macros.conf` and the skeletal `/etc/httpd/conf.d/001-events.conf` from an existing server. Make sure the following lines are in httpd.conf:
120 134
... ...
@@ -134,6 +148,12 @@ Instead of having the `ANDROID_HOME` environment variable be set in `/etc/profil
134 148
135 149
Had to fiddle a little with the JDK being used. The default installation has an OpenJDK installed, and the AWS tools depend on it. Therefore, it cannot just be removed. As a result, it's important that `env.sh` has the correct `JAVA_HOME` set (/opt/jdk1.8.0_45, in this case). Otherwise, the OSGi environment won't properly start up.
136 150
151
+For the ``root`` user create the symbolic link from ``/root/crontab`` to ``/home/sailing/code/configuration/crontab`` and run ``crontab crontab``. It adds the following crontab entry that is responsible for updating the SSH keys of the users with permission for landscape management in the ``/root/.ssh/authorized_keys`` file.
152
+```
153
+* * * * * export PATH=/bin:/usr/bin:/usr/local/bin; sleep $(( $RANDOM * 60 / 32768 )); update_authorized_keys_for_landscape_managers_if_changed $( cat /root/ssh-key-reader.token ) https://security-service.sapsailing.com /root 2>&1 >>/var/log/sailing.err
154
+```
155
+Make sure, a valid bearer token is installed in ``/root/ssh-key-reader.token``.
156
+
137 157
To ensure that chronyd is started during the boot sequence, issued the command
138 158
139 159
```