Tag: Apache

OpenProject Apache reverse proxy with https secure connection

These are some notes on setting up OpenProject on a backend server (let’s call it backsrv.example.com), and accessing it via a front-end system (frontsrv.example.com). Normally we’d do the SSL termination at the reverse proxy, and there is some documentation on this. In this case I wanted to do things properly, and protect the login credentials all the way. This means using an https connection between the reverse proxy and the back end server.

Firstly, the reverse proxy has to trust the SSL certificate that the back end uses. There are several ways to go about this. I chose to set up a local certificate authority using the easy-rsa scripts (using another small virtual machine set up only for this purpose). For one connection this is probably overkill, but for multiple backends in the future it will make the administration a lot easier.

  • Set up CA
    • Debian 10, install easy-rsa package, do required setup.
  • Copy CA root certificate to frontsrv
    • For Debian systems, copy to /usr/local/share/ca-certificates/ and run update-ca-certificates
  • Create CSR on backsrv, copy it to CA, sign it and copy resulting certificate to backsrv. Put cert and key in sensible places (/etc/ssl/private/ and /etc/ssl/local-certs/). Make sure permissions are correct.
  • Configure Apache on backsrv and check cert works (for OpenProject edit /etc/openproject/installer.dat to put in the correct certificate paths and run openproject configure to update the config).

Set up Apache to do proxy stuff on frontsrv. Here’s the beginning fragment of default-ssl.conf that should work:

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerAdmin webmaster@localhost

                DocumentRoot /var/www/html

                RequestHeader edit Destination ^https http early

                SSLProxyEngine on
                SSLProxyCheckPeerName off

                # To openproject server on backserv
                ProxyPass /openproject https://backsrv.example.com/openproject
                ProxyPassReverse /openproject https://backsrv.example.com/openproject
                <Location /openproject>
                        ProxyPreserveHost On
                        Require all granted
                </Location>

You also need to go to the OpenProject web interface admin area, go to System Settings – General and change the Host name to the reverse proxy, and set protocol to https. It will complain if there’s a hostname mismatch (case sensitive, even!). You may also want to go to Email – Email notifications and change the Emission email address to be consistent.

Don’t forget, need SSLProxyEngine on!

For OpenProject the subdirectory locations on the front and back ends do need to match.

The ProxyPreserveHost On is required per the OpenProject documentation. Unfortunately, that means it tries to match the name frontsrv.example.com to the back end cert, and the SSL handshake fails. This is the reason for the SSLProxyCheckPeerName off directive – it disables checking the certificate CN or Subject Alternative Names.

Apparently the SSLProxyCheckPeerName off can go in a <Proxy>...</Proxy> matching block with Apache 2.4.30 or newer, which would be nice. As it is this will turn it off for the whole vhost, which is a small lessening of security.

I suppose in principle we could create the certificate for the back end with the name of the front end, or add it to the SANs. I haven’t tried this and it seems like it could be a recipe for confusion and subtle bugs.

Publishing websites with Jekyll, Apache and SVN

Now I’ve got this working to some extent here are some notes about setting up Jekyll with SVN and Apache:

Server – Debian 9 Stretch, normal command-line only install. Set up system to use email server (campus smarthost in our case).

Install SVN and Apache and set up accordingly.

Install Jekyll:

apt install jekyll

Create an SVN repository for the site files.

Create new project directories at a temporary location, e.g.

jekyll new /tmp/newsite

Commit these files to the SVN repository (I normally check out the repository on my local workstation, copy the directory in /tmp from the server into the working directory on the workstation, add them and commit). Delete the directory in /tmp.

On the server, create the actual website file location by exporting from the SVN via a temporary location:

svn export file:///path/to/repository /tmp/buildfiles
jekyll build --source /tmp/buildfiles /var/www/sitename
rm -Rf /tmp/buildfiles

Configure Apache to serve from /var/www/sitename. In our case we ultimately wanted to serve multiple sites through a reverse proxy, so we used a vhost serving on an alternate port. This can be a handy testing configuration – you don’t have to worry about fiddling with the other website settings. For example, using port 8081:

<VirtualHost *:8081>

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/sitename

</VirtualHost>

(Remember to change ports.conf to listen on the new port!)

Test by pointing a webserver at server:8081

Once that’s all working, set up the post-commit hook script to automatically build the site on a commit. Our current setup is:

#!/bin/sh

# POST-COMMIT HOOK

REPOS="$1"
REV="$2"
REPOS_BASENAME=$(/usr/bin/basename "$REPOS")
TMP_SVN_EXPORT="/tmp/$REPOS_BASENAME"

# These two need configured!
PUBLIC_WWW="/var/www/sitename"
BUILD_EMAIL="your.email@this.address"

"$REPOS"/hooks/mailer.py commit "$REPOS" $REV "$REPOS"/hooks/mailer.conf

LOGVAR=$(/export0/svn_config/jekyll_build.sh "$REPOS" $REV "$TMP_SVN_EXPORT" "$PUBLIC_WWW" 2>&1)

echo "$LOGVAR" | /usr/bin/unix2dos | mail -s "$REPOS_BASENAME build $REV" "$BUILD_EMAIL"

(Note that on Debian you need to install the dos2unix package. Needed as plain text email expects CRLF line terminators as specified in RFC 2822.)

#!/bin/sh

REPOS="$1"
REV="$2"
TMP_SVN_EXPORT="$3"
PUBLIC_WWW="$4"

/usr/bin/svn export --quiet file:///"$REPOS" "$TMP_SVN_EXPORT"
/usr/bin/jekyll build --source "$TMP_SVN_EXPORT" --destination "$PUBLIC_WWW"
rm -Rf "$TMP_SVN_EXPORT"

Note that the build process runs under the Apache user account, so set permissions appropriately. Also, when troubleshooting remember that on Debian 9 the Apache process is configured by default to use a private /tmp directory!

This works for our current needs, although it isn’t optimised. Improvements would be:

  • Unify the setup for the commit email and build email scripts.
  • Build the site in the background (although you’d have to tweak how the logging output works in that case).

Of course, the professionals would use something like a combination of GitLab and Jenkins to automate this stuff properly…

 

Private /tmp directories in Debian 9 Stretch with Apache

In Debian 9 Stretch Apache is configured to use systemd‘s PrivateTmp feature by default. This means that the Apache tmp directory actually lives in /tmp/systemd-private-BIGLONGSTRING--apache2.service-STRING.

So if you are running an SVN server that uses Apache for serving, anything written to /tmp in the hook scripts ends up in the private directory rather than the normal userspace one.

502 Bad Gateway with SVN copy, move or rename behind a reverse proxy

Problem: You have a SVN server sitting behind a reverse web proxy (e.g. for convenient SSL termination purposes). This works for new files, changes etc. but fails when you try to rename something, make a copy or move. The error is:

Unexpected HTTP status 502 'Bad Gateway'

The reason is explained here, but to summarise:

These operations involve sending a http COPY method. This includes a Destination: field in the header, which is not rewritten by Apache’s ProxyPass directives. Thus the destination field starts with https – not http. This confuses mod_dav on the SVN server.

The solution is to change the header. We can do this on Apache (2.2 or higher) using the headers module. This can be done either on the proxy server or the SVN server. As my SVN server is very old (the main reason why it’s behind a proxy) I’ll do this on the proxy server.

Enable the headers module if required. On Debian:

# a2enmod headers

and restart Apache. Then alter your configuration to include:

RequestHeader edit Destination ^https http early

This probably should go before any ProxyPass directives.

Then your config might look like:

RequestHeader edit Destination ^https http early

ProxyPass /svn http://your.real.svn.server/svn
ProxyPassReverse /svn http://your.real.svn.server/svn
<Location /svn/>
  Require all granted
</Location>

(Note this is for a gateway system where other locations can proxy to other application servers.)

ResourceSpace cron and database notes

Ran into a couple of issues today:

Note: system setup is Debian 9 with standard options (Apache 2.4, PHP 7.0, MariaDB 10.1)

Cron

The documentation implies you should run cron_copy_hitcount.php as a cron job. However, the new correct way seems to be to run batch/cron.php, which runs a bunch of sub-jobs. I’ve got this set up in cron.daily as:

#!/bin/sh
wget -q -r http://localhost/resourcespace/batch/cron.php

We’ll see if this works. Certainly running it directly by browsing to it seems to work.

LDAP

Trying to activate the simpleldap plugin threw up two problems:

php-ldap wasn’t installed – easy enough. Note apache needs a restart after installing…

Second error was a problem with the database – the plugin couldn’t create a table, with error

Specified key was too long; max key length is 767 bytes

This seems to be because when I created the database the character set used was utf8mb4_general_ci, which in the worst case uses 4 bytes per character. If you try to create a index key with 255 characters you run into this limit.

The solution was to change the database to use utf8_general_ci. This allowed the plugin to create the simpleldap_groupmap table with utf8_general_ci. The rest of the database is still utf8mb4_general_ci, but as it has been created already without an issue we should be ok.

Using custom Diffie-Hellman parameters with Apache 2.2.22 and OpenSSL 1.0.1e (Debian 7 Wheezy)

See https://weakdh.org for the problem – 1024 bit Diffie-Hellman keys are potentially breakable (the ‘logjam’ vulnerability). This can be fixed in Apache 2.4 by pointing it at a custom key, but up to recently ver 2.2 was vulnerable. The issue was fixed in apache 2.2.22-13+deb7u5, which allows a custom DH key to be appended to the server certificate. To use this in Debian 7:

Update to apache 2.2.22-13+deb7u5 or higher.

Generate a new Diffie-Hellman group using

openssl dhparam -out dhparams.pem 2048

Find where the appropriate server certificate file is – standard debian setup specifies this in

/etc/apache2/sites-available/default-ssl

Append the DH group to the server certificate

cat dhparams.pem >> server_certificate.pem

The resulting file should look like

-----BEGIN CERTIFICATE-----
stuff
-----END CERTIFICATE-----
-----BEGIN DH PARAMETERS-----
more stuff
-----END DH PARAMETERS-----

Restart Apache.

Checking this using the https://www.ssllabs.com/ssltest/ shows DH 2048 bits