OpenProject Apache reverse proxy with https secure connection

These are some notes on setting up OpenProject on a backend server (let’s call it backsrv.example.com), and accessing it via a front-end system (frontsrv.example.com). Normally we’d do the SSL termination at the reverse proxy, and there is some documentation on this. In this case I wanted to do things properly, and protect the login credentials all the way. This means using an https connection between the reverse proxy and the back end server.

Firstly, the reverse proxy has to trust the SSL certificate that the back end uses. There are several ways to go about this. I chose to set up a local certificate authority using the easy-rsa scripts (using another small virtual machine set up only for this purpose). For one connection this is probably overkill, but for multiple backends in the future it will make the administration a lot easier.

  • Set up CA
    • Debian 10, install easy-rsa package, do required setup.
  • Copy CA root certificate to frontsrv
    • For Debian systems, copy to /usr/local/share/ca-certificates/ and run update-ca-certificates
  • Create CSR on backsrv, copy it to CA, sign it and copy resulting certificate to backsrv. Put cert and key in sensible places (/etc/ssl/private/ and /etc/ssl/local-certs/). Make sure permissions are correct.
  • Configure Apache on backsrv and check cert works (for OpenProject edit /etc/openproject/installer.dat to put in the correct certificate paths and run openproject configure to update the config).

Set up Apache to do proxy stuff on frontsrv. Here’s the beginning fragment of default-ssl.conf that should work:

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerAdmin webmaster@localhost

                DocumentRoot /var/www/html

                RequestHeader edit Destination ^https http early

                SSLProxyEngine on
                SSLProxyCheckPeerName off

                # To openproject server on backserv
                ProxyPass /openproject https://backsrv.example.com/openproject
                ProxyPassReverse /openproject https://backsrv.example.com/openproject
                <Location /openproject>
                        ProxyPreserveHost On
                        Require all granted
                </Location>

You also need to go to the OpenProject web interface admin area, go to System Settings – General and change the Host name to the reverse proxy, and set protocol to https. It will complain if there’s a hostname mismatch (case sensitive, even!). You may also want to go to EmailEmail notifications and change the Emission email address to be consistent.

Don’t forget, need SSLProxyEngine on!

For OpenProject the subdirectory locations on the front and back ends do need to match.

The ProxyPreserveHost On is required per the OpenProject documentation. Unfortunately, that means it tries to match the name frontsrv.example.com to the back end cert, and the SSL handshake fails. This is the reason for the SSLProxyCheckPeerName off directive – it disables checking the certificate CN or Subject Alternative Names.

Apparently the SSLProxyCheckPeerName off can go in a <Proxy>...</Proxy> matching block with Apache 2.4.30 or newer, which would be nice. As it is this will turn it off for the whole vhost, which is a small lessening of security.

I suppose in principle we could create the certificate for the back end with the name of the front end, or add it to the SANs. I haven’t tried this and it seems like it could be a recipe for confusion and subtle bugs.

Making nis authentication work with Ubuntu 16, Debian 8, Fedora 28 etc.

After updating anything to use systemd-235 NIS logins either don’t work at all (usually for GUI logins), or take a long time to login (console or ssh, sometimes). The culprit is a line in the systemd-logind.service:

IPAddressDeny=any

This sandboxes the service and doesn’t allow it to talk to the network. Unfortunately this affects nis lookups done via the glibc NSS API. See the links at https://github.com/systemd/systemd/pull/7343

The quick solution is to turn off the sandboxing, either by commenting out or changing the line in systemd-logind.service, or creating a drop-in snippet that overrides it. This can be done by creating a file /etc/systemd/system/systemd-logind.service.d/IPAddress_clear.conf with the contents:

[Service]
IPAddressDeny=

The file can be called anything you like (.conf).

Then restart things:

systemctl daemon-reload
systemctl restart systemd-logind.service

You can check that the drop-in is being loaded with

systemctl status systemd-logind.service

In the output you should see something like:

   Loaded: loaded (/lib/systemd/system/systemd-logind.service; static; vendor preset: enabled)
  Drop-In: /etc/systemd/system/systemd-logind.service.d
  └─IPAddress_clear.conf

The other test is to see if NIS logins work correctly, of course…

The slightly slower solution is to use nscd to cache the lookup requests, and apparently does so in a way that plays nicely with the sandboxing. The much slower solution is to switch to using sssd or similar and ditch NIS once and for all…

Note – this may also affect systemd-udevd.

Invalid self-signed certificates in FreeNAS 11

Had a problem with several FreeNAS 11 systems – when trying to enable https for the web interface I couldn’t get a working certificate.

  1. Generate internal CA.
  2. Generate internal certificate.
  3. Chrome complains the certificate is invalid and does not allow you to bypass it.
  4. Delete certificate and CA, repeat, same problem.

After mucking about with using upper and lower case for the certificate CN names etc, eventually got things to work by generating a second certificate with the same CA. Which makes me think it might be this problem:

FreeNAS-Generated Certificates: Buyer Beware

So either generating a couple of certificates (and using the second one) or changing the serial number of the CA to something bigger than 1 (and then generating a certificate) should work.

(Go to the CA tab, double-click the CA entry and scroll down to the bottom to view/edit the serial number).

Setting proxy in Windows Server 2016 for the Update service

In Windows Server 2016 the usual trick of just setting a proxy server in IE doesn’t seem to work. The proxy can be set from the command line, but there is some contradictory advice out there. What worked for me was:

netsh winhttp set proxy proxy-server="your.proxy.server:port" bypass-list="*.your.local.domain"

This should result in:

Current WinHTTP proxy settings:

    Proxy Server(s) : your.proxy.server:port
    Bypass List : *.your.local.domain

You can also check this with

netsh winhttp show proxy

You are meant to be able to import this from IE, but that gave me a syntax error.

You may need to restart the update service after this to kick it into life.

User missing from login screen – OSX with FileVault

Situation: new MacBook with OSX Sierra. Set up with an admin account, enable FileVault (taking note of recovery key obviously!) and install the necessary. Create account for end user and give it to them. All is well (after getting some USB-A to USB-C converters…)

User restores all his stuff from a Time Machine backup to the account on the new system – this overwrites all the current user settings. After rebooting the system, his account has disappeared from the login screen.

Solution: Log on as the other administrative user (luckily we have one!) and open the Settings – Security & Privacy – FileVault. A notice at the bottom of the dialog box appears informing you that there are some users that are not enabled to use FileVault, with a button to enable the users. This brings up a list showing the missing user. To enable the user their password needs to be entered.

Using custom Diffie-Hellman parameters with Apache 2.2.22 and OpenSSL 1.0.1e (Debian 7 Wheezy)

See https://weakdh.org for the problem – 1024 bit Diffie-Hellman keys are potentially breakable (the ‘logjam’ vulnerability). This can be fixed in Apache 2.4 by pointing it at a custom key, but up to recently ver 2.2 was vulnerable. The issue was fixed in apache 2.2.22-13+deb7u5, which allows a custom DH key to be appended to the server certificate. To use this in Debian 7:

Update to apache 2.2.22-13+deb7u5 or higher.

Generate a new Diffie-Hellman group using

openssl dhparam -out dhparams.pem 2048

Find where the appropriate server certificate file is – standard debian setup specifies this in

/etc/apache2/sites-available/default-ssl

Append the DH group to the server certificate

cat dhparams.pem >> server_certificate.pem

The resulting file should look like

-----BEGIN CERTIFICATE-----
stuff
-----END CERTIFICATE-----
-----BEGIN DH PARAMETERS-----
more stuff
-----END DH PARAMETERS-----

Restart Apache.

Checking this using the https://www.ssllabs.com/ssltest/ shows DH 2048 bits

A nice way of configuring linux system-wide login notifications using PAM

There are several ways to have a linux box email you when someone logs in to it. Most of these use a script in either the local profile files (for individual users) or the system-wide profile (and/or in sshrc). Another nice way is to use the pam authentication system to do the job. A setup is given at:

http://blog.stalkr.net/2010/11/login-notifications-pamexec-scripting.html

Brief setup

Assuming Debian here.

Make sure the system is set up to talk to an email server and has some way of sending emails from the command line. The script here assumes the mailx package is installed.

Quick test:

echo "test" | mail -s "test" user@example.com

Create a script somewhere sensible (e.g. /usr/local/bin) and make it executable.

#!/bin/sh
[ "$PAM_TYPE" = "open_session" ] || exit 0
{
echo "User: $PAM_USER"
echo "Ruser: $PAM_RUSER"
echo "Rhost: $PAM_RHOST"
echo "Service: $PAM_SERVICE"
echo "TTY: $PAM_TTY"
echo "Date: `date`"
echo "Server: `uname -a`"
} | mail -s "`hostname -s` $PAM_SERVICE login: $PAM_USER" user@example.com

replacing user@example.com with the required email address. Note that you could send the email to root or another local account if you have that aliased to an external address.

edit /etc/pam.d/common-session to add at the end:

session optional pam_exec.so /usr/local/bin/login_notify

and test it by logging in with an ssh session (terminal or winscp etc.)

Sudo uses common-session-noninteractive – not sure whether it would be best to put this in /etc/pam.d/sudo as well…

Debian 6 (squeeze) to 7 (wheezy) upgrade notes

Networking (firewall related)

Some of the older debian servers are firewalled using a command in the ”/etc/network/interfaces” file to load the iptables rules. For some reason this breaks the file on upgrade to Debian wheezy and the network interfaces don’t come up on boot. To fix, comment or remove the line:

pre-up iptables-restore < /etc/iptables.conf

and use the iptables-persistent package instead.

Also gives an error on

INPUT -i ! lo

Need to change to

INPUT ! -i lo

(To be fair, this has been deprecated for a while now)

Suhosin

php5-suhosin has been removed from wheezy. The upgrade sort of removes it, but it hangs around as a remove candidate package. Following http://pc-freak.net/blog/how-to-get-rid-of-php-warning-php-startup-unable-to-load-dynamic-library-usrlibphp520090626suhosin-so-on-debian-gnu-linux/ we check the package status with

dpkg -l | grep -i suhosin

and remove with

dpkg --purge php5-suhosin

This seems to fix the problem (don’t get emailed every so often by cron about it missing)

For next upgrade should remove it first.

Kernel

Make sure the generic kernel package is installed – have tended to choose the specific one in the past for no good reason.

apt-get install linux-image-amd64

This should ensure the 3.2 kernel is installed as part of the upgrade.