Changes in 65f3993: Created creating ec2 image for webserver from scratch (markdown)

				wiki/info/landscape/creating-ec2-image-for-webserver-from-scratch.md
			
          @@ -0,0 +1,133 @@

          +# Setting up an image for the www.sapsailing.com web server

          +

          +This is an add-on to the regular EC2 image set-up described [here](https://wiki.sapsailing.com/wiki/info/landscape/creating-ec2-image-from-scratch), but leave out the following packages during installation because they are not needed on the webserver:

          +

          +* libstdc++48.i686 (for Android builds)

          +* glibc.i686 (for Android builds)

          +* libzip.i686 (for Android builds)

          +* telnet

          +* chrony (ntp is used now instead)

          +

          +Then carry out these steps:

          +

          +* install additional packages: `yum install git mod24_perl perl perl-CGI perl-Template-Toolkit perl-HTML-Template perl-CPAN perl-DBD-MySQL mod24_ssl php71 mod24-ldap ruby24 ruby24-devel rubygems24 rubygems24-devel icu libicu-devel gcc-c++ ncurses-devel geoip-devel`

          +* run the following command in order to obtain this feature required by Bugzilla:

          +```

          +cpan install Date::Parse Email::Address Email::Send DBI

          +```

          +* Since goaccess seems to have been removed from the Amazon Linux repositories, compile it from sources. See [https://goaccess.io/download](https://goaccess.io/download)

          +* make sure `/etc/alternatives/ruby` and `/etc/alternatives/gem` point to `/usr/bin/[ruby|gem]2.4`

          +* run the following commands to install gollum and uninstall a too current rack version 2.0.3:

          +```

          +gem install gollum

          +gem uninstall rack

          +Select gem to uninstall:

          + 1. rack-1.6.8

          + 2. rack-2.0.3

          + 3. All versions

          +> 2

          +

          +You have requested to uninstall the gem:

          +        rack-2.0.3

          +

          +sinatra-2.0.0 depends on rack (~> 2.0)

          +If you remove this gem, these dependencies will not be met.

          +Continue with Uninstall? [yN]  y

          +Successfully uninstalled rack-2.0.3

          +

          +```

          +* ensure there are users and groups for `wiki`, `scores`, `wordpress`, `trac` that match up with their /home directory owners / groups

          +* ensure the Wiki startup script `serve.sh` configured for port 4567 and `config.ru` as well as the entire Gollum installation under /home/wiki are present, as well as the `users.yml` file

          +* ensure there is a reasonable `/root/.goaccess` file

          +* ensure there is the `/etc/tmux.conf` file that maps your hotkeys (Ctrl-a vs. Ctrl-b, for example)

          +* install bugzilla to `/usr/share/bugzilla` and `/var/lib/bugzilla`

          +* create `/etc/bugzilla/localconfig`

          +* set up crontab for user `wiki` as `*/10 * * * * /home/wiki/syncgit` and make sure the script is in place

          +* comment `lbmethod_heartbeat_module` in /etc/httpd/conf.modules.d/00-proxy.conf because we don't need this sort of load balancing across origin servers and it causes a warning message in error_log

          +* (NOT WORKING CURRENTLY BECAUSE OF SINGLE-THREADED PHP: in `/etc/httpd/conf.module.d/00-mpm.conf` select `worker` instead of `prefork`)

          +* install awstats to `/usr/share/awstats`, establish `/etc/httpd/conf/passwd.awstats` and create /etc/cron.weekly/awstats as follows:

          +```

          +#!/bin/bash

          +su -l -c '/usr/share/awstats/tools/awstats_updateall.pl now         -configdir="/etc/awstats"         -awstatsprog="/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" >>/var/log/awstats-cron.out 2>>/var/log/awstats-cron.err'

          +#exec /usr/share/awstats/tools/awstats_updateall.pl now         -configdir="/etc/awstats"         -awstatsprog="/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" >>/var/log/awstats-cron.out 2>>/var/log/awstats-cron.err

          +exit 0

          +```

          +* Follow the [mail setup](https://wiki.sapsailing.com/wiki/info/landscape/mail-relaying#setup-central-mail-server-instance-webserver) instructions

          +* Install the backup.sh script with the following contents:

          +```

          +#!/bin/sh

          +

          +# This is a template for creating backup of data and persist them

          +# to a central backup server. Make sure to adapt this script to your needs.

          +#

          +# Maintainer: simon.marcel.pamies@sap.com

          +

          +export HOME=/root

          +

          +# Directories to backup

          +BACKUP_DIRECTORIES="/etc /home/trac/git /home/trac/mailinglists /home/trac/maven-repositories /home/trac/p2-repositories /home/trac/releases /home/trac/sapsailing_layouts.git /var/www/static /home/trac/crontab /home/scores /var/log/old"

          +

          +# Prefix for backup - set that to the hostname of your server

          +# Make sure to change this!

          +PREFIX="webserver"

          +

          +# Directory for temporary files

          +TARGET_DIR=/tmp

          +

          +# Configuration for external backup server - needs a ssh key

          +BACKUP_SERVER="backup@172.31.25.136"

          +

          +# Set date for this backup - this makes it possible to compare

          +# files from different branches

          +BACKUP_DATE=`date +%s`

          +

          +# Aliases

          +BUP_CMD="/opt/bup/bup"

          +BUP_ADDITIONAL="-r $BACKUP_SERVER:/home/backup/$PREFIX"

          +BUP_IGNORES='--exclude-rx=/war/$ --exclude-rx=/cache/unique-ips-per-referrer/stats/'

          +BUP_CMD_INDEX="$BUP_CMD index $BUP_IGNORES"

          +BUP_CMD_SAVE="$BUP_CMD save --date=$BACKUP_DATE $BUP_ADDITIONAL"

          +

          +PARAM=$@

          +

          +# Make sure to init remote repository

          +$BUP_CMD init

          +ssh $BACKUP_SERVER "/opt/bup/bup init -r /home/backup/$PREFIX"

          +

          +# Backup general directories

          +for OTHER_DIR in $BACKUP_DIRECTORIES; do

          +        NORMALIZED_DIR_NAME=${OTHER_DIR//\//-}

          +        $BUP_CMD_INDEX $OTHER_DIR

          +        $BUP_CMD_SAVE -n dir$NORMALIZED_DIR_NAME $OTHER_DIR

          +        # no need to remove any backup files here

          +done

          +```

          +* Register the backup script as a cron job by establishing the following `/etc/crontab` file:

          +```

          +SHELL=/bin/bash

          +PATH=/sbin:/bin:/usr/sbin:/usr/bin

          +MAILTO=simon.marcel.pamies@sap.com

          +HOME=/

          +

          +# For details see man 4 crontabs

          +

          +# Example of job definition:

          +# .---------------- minute (0 - 59)

          +# |  .------------- hour (0 - 23)

          +# |  |  .---------- day of month (1 - 31)

          +# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...

          +# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat

          +# |  |  |  |  |

          +# *  *  *  *  * user-name command to be executed

          +0 22 * * * root /opt/backup.sh

          +```

          +* Install Wordpress

          +* Install gollum Wiki

          +* Copy git contents of ssh://trac@sapsailing.com/home/trac/git to /home/trac/git

          +* Ensure there is a /home/scores directory with subdirectories `barbados`, `kiwo`, `sailwave`, `scores`, `velum`, and `xrrftp`.

          +* Establish the Apache web server configuration, in particular ensure that the SSL certificates are in place (see [here](https://wiki.sapsailing.com/wiki/info/security/ssl-support)) and the following files are set up: `/etc/httpd/conf/httpd.conf`, `/etc/httpd/conf/passwd.awstats`, `/etc/httpd/conf/passwd.git`, and `/etc/httpd/conf/conf.d/*.conf`.

          +

          +

          +## Appendix / Resources

          +BACKUP_DIRECTORIES="/etc /home/trac/git /home/trac/mailinglists /home/trac/maven-repositories /home/trac/p2-repositories /home/trac/releases /home/trac/sapsailing_layouts.git /var/www/static /home/trac/crontab /home/scores /var/log/old"

          +

...	...	@@ -0,0 +1,133 @@
	1	+# Setting up an image for the www.sapsailing.com web server
	2	+
	3	+This is an add-on to the regular EC2 image set-up described [here](https://wiki.sapsailing.com/wiki/info/landscape/creating-ec2-image-from-scratch), but leave out the following packages during installation because they are not needed on the webserver:
	4	+
	5	+* libstdc++48.i686 (for Android builds)
	6	+* glibc.i686 (for Android builds)
	7	+* libzip.i686 (for Android builds)
	8	+* telnet
	9	+* chrony (ntp is used now instead)
	10	+
	11	+Then carry out these steps:
	12	+
	13	+* install additional packages: `yum install git mod24_perl perl perl-CGI perl-Template-Toolkit perl-HTML-Template perl-CPAN perl-DBD-MySQL mod24_ssl php71 mod24-ldap ruby24 ruby24-devel rubygems24 rubygems24-devel icu libicu-devel gcc-c++ ncurses-devel geoip-devel`
	14	+* run the following command in order to obtain this feature required by Bugzilla:
	15	+```
	16	+cpan install Date::Parse Email::Address Email::Send DBI
	17	+```
	18	+* Since goaccess seems to have been removed from the Amazon Linux repositories, compile it from sources. See [https://goaccess.io/download](https://goaccess.io/download)
	19	+* make sure `/etc/alternatives/ruby` and `/etc/alternatives/gem` point to `/usr/bin/[ruby\|gem]2.4`
	20	+* run the following commands to install gollum and uninstall a too current rack version 2.0.3:
	21	+```
	22	+gem install gollum
	23	+gem uninstall rack
	24	+Select gem to uninstall:
	25	+ 1. rack-1.6.8
	26	+ 2. rack-2.0.3
	27	+ 3. All versions
	28	+> 2
	29	+
	30	+You have requested to uninstall the gem:
	31	+ rack-2.0.3
	32	+
	33	+sinatra-2.0.0 depends on rack (~> 2.0)
	34	+If you remove this gem, these dependencies will not be met.
	35	+Continue with Uninstall? [yN] y
	36	+Successfully uninstalled rack-2.0.3
	37	+
	38	+```
	39	+* ensure there are users and groups for `wiki`, `scores`, `wordpress`, `trac` that match up with their /home directory owners / groups
	40	+* ensure the Wiki startup script `serve.sh` configured for port 4567 and `config.ru` as well as the entire Gollum installation under /home/wiki are present, as well as the `users.yml` file
	41	+* ensure there is a reasonable `/root/.goaccess` file
	42	+* ensure there is the `/etc/tmux.conf` file that maps your hotkeys (Ctrl-a vs. Ctrl-b, for example)
	43	+* install bugzilla to `/usr/share/bugzilla` and `/var/lib/bugzilla`
	44	+* create `/etc/bugzilla/localconfig`
	45	+* set up crontab for user `wiki` as `/10 * * * /home/wiki/syncgit` and make sure the script is in place
	46	+* comment `lbmethod_heartbeat_module` in /etc/httpd/conf.modules.d/00-proxy.conf because we don't need this sort of load balancing across origin servers and it causes a warning message in error_log
	47	+* (NOT WORKING CURRENTLY BECAUSE OF SINGLE-THREADED PHP: in `/etc/httpd/conf.module.d/00-mpm.conf` select `worker` instead of `prefork`)
	48	+* install awstats to `/usr/share/awstats`, establish `/etc/httpd/conf/passwd.awstats` and create /etc/cron.weekly/awstats as follows:
	49	+```
	50	+#!/bin/bash
	51	+su -l -c '/usr/share/awstats/tools/awstats_updateall.pl now -configdir="/etc/awstats" -awstatsprog="/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" >>/var/log/awstats-cron.out 2>>/var/log/awstats-cron.err'
	52	+#exec /usr/share/awstats/tools/awstats_updateall.pl now -configdir="/etc/awstats" -awstatsprog="/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" >>/var/log/awstats-cron.out 2>>/var/log/awstats-cron.err
	53	+exit 0
	54	+```
	55	+* Follow the [mail setup](https://wiki.sapsailing.com/wiki/info/landscape/mail-relaying#setup-central-mail-server-instance-webserver) instructions
	56	+* Install the backup.sh script with the following contents:
	57	+```
	58	+#!/bin/sh
	59	+
	60	+# This is a template for creating backup of data and persist them
	61	+# to a central backup server. Make sure to adapt this script to your needs.
	62	+#
	63	+# Maintainer: simon.marcel.pamies@sap.com
	64	+
	65	+export HOME=/root
	66	+
	67	+# Directories to backup
	68	+BACKUP_DIRECTORIES="/etc /home/trac/git /home/trac/mailinglists /home/trac/maven-repositories /home/trac/p2-repositories /home/trac/releases /home/trac/sapsailing_layouts.git /var/www/static /home/trac/crontab /home/scores /var/log/old"
	69	+
	70	+# Prefix for backup - set that to the hostname of your server
	71	+# Make sure to change this!
	72	+PREFIX="webserver"
	73	+
	74	+# Directory for temporary files
	75	+TARGET_DIR=/tmp
	76	+
	77	+# Configuration for external backup server - needs a ssh key
	78	+BACKUP_SERVER="backup@172.31.25.136"
	79	+
	80	+# Set date for this backup - this makes it possible to compare
	81	+# files from different branches
	82	+BACKUP_DATE=`date +%s`
	83	+
	84	+# Aliases
	85	+BUP_CMD="/opt/bup/bup"
	86	+BUP_ADDITIONAL="-r $BACKUP_SERVER:/home/backup/$PREFIX"
	87	+BUP_IGNORES='--exclude-rx=/war/$ --exclude-rx=/cache/unique-ips-per-referrer/stats/'
	88	+BUP_CMD_INDEX="$BUP_CMD index $BUP_IGNORES"
	89	+BUP_CMD_SAVE="$BUP_CMD save --date=$BACKUP_DATE $BUP_ADDITIONAL"
	90	+
	91	+PARAM=$@
	92	+
	93	+# Make sure to init remote repository
	94	+$BUP_CMD init
	95	+ssh $BACKUP_SERVER "/opt/bup/bup init -r /home/backup/$PREFIX"
	96	+
	97	+# Backup general directories
	98	+for OTHER_DIR in $BACKUP_DIRECTORIES; do
	99	+ NORMALIZED_DIR_NAME=${OTHER_DIR//\//-}
	100	+ $BUP_CMD_INDEX $OTHER_DIR
	101	+ $BUP_CMD_SAVE -n dir$NORMALIZED_DIR_NAME $OTHER_DIR
	102	+ # no need to remove any backup files here
	103	+done
	104	+```
	105	+* Register the backup script as a cron job by establishing the following `/etc/crontab` file:
	106	+```
	107	+SHELL=/bin/bash
	108	+PATH=/sbin:/bin:/usr/sbin:/usr/bin
	109	+MAILTO=simon.marcel.pamies@sap.com
	110	+HOME=/
	111	+
	112	+# For details see man 4 crontabs
	113	+
	114	+# Example of job definition:
	115	+# .---------------- minute (0 - 59)
	116	+# \| .------------- hour (0 - 23)
	117	+# \| \| .---------- day of month (1 - 31)
	118	+# \| \| \| .------- month (1 - 12) OR jan,feb,mar,apr ...
	119	+# \| \| \| \| .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
	120	+# \| \| \| \| \|
	121	+# * * * * * user-name command to be executed
	122	+0 22 * * * root /opt/backup.sh
	123	+```
	124	+* Install Wordpress
	125	+* Install gollum Wiki
	126	+* Copy git contents of ssh://trac@sapsailing.com/home/trac/git to /home/trac/git
	127	+* Ensure there is a /home/scores directory with subdirectories `barbados`, `kiwo`, `sailwave`, `scores`, `velum`, and `xrrftp`.
	128	+* Establish the Apache web server configuration, in particular ensure that the SSL certificates are in place (see [here](https://wiki.sapsailing.com/wiki/info/security/ssl-support)) and the following files are set up: `/etc/httpd/conf/httpd.conf`, `/etc/httpd/conf/passwd.awstats`, `/etc/httpd/conf/passwd.git`, and `/etc/httpd/conf/conf.d/*.conf`.
	129	+
	130	+
	131	+## Appendix / Resources
	132	+BACKUP_DIRECTORIES="/etc /home/trac/git /home/trac/mailinglists /home/trac/maven-repositories /home/trac/p2-repositories /home/trac/releases /home/trac/sapsailing_layouts.git /var/www/static /home/trac/crontab /home/scores /var/log/old"
	133	+