Running svnadmin verify as root permission issues on Berkeley DB repositories

After a system crash I ran svnadmin verify on some of the relevant repositories to check things were ok – this was done as root. Afterwards normal network access (via Apache) was broken with

Internal error: Berkeley DB error for filesystem

appearing in the logs. The fix was to chown -R www-data:www-data on the broken repositories. It looks like running svnadmin verify as root changes permissions on SVN repositories somewhere (at least Berkeley DB ones).

Lesson learned – check normal SVN access after doing this sort of thing!

ANSI control codes in Jekyll output breaking emails

Also see Publishing websites with Jekyll, Apache and SVN

If you send console output via email (like, say the output of jekyll build as part of a SVN post-commit hook script) if there are ANSI control characters in the string (e.g. colour codes) this can break things. In this case the mail command (Debian 9 default exim) was only sending text up to the first ANSI code, which meant that the jekyll build error messages (which are yellow and red) were missing.

To fix this pipe the text through ansi2txt (comes with the colorized-logs package in Debian and Ubuntu). This strips out all ANSI control codes making the string email safe.

(After this I pipe it through unix2dos to convert to CRLF line endings, as this appears to be the standard for email. On Debian this comes with the dos2unix package.)

The last line in the hook script then becomes

echo "$LOGVAR" | /usr/bin/ansi2txt | /usr/bin/unix2dos | mail -s "$REPOS_BASENAME build $REV" "$BUILD_EMAIL"

Publishing websites with Jekyll, Apache and SVN

Now I’ve got this working to some extent here are some notes about setting up Jekyll with SVN and Apache:

Server – Debian 9 Stretch, normal command-line only install. Set up system to use email server (campus smarthost in our case).

Install SVN and Apache and set up accordingly.

Install Jekyll:

apt install jekyll

Create an SVN repository for the site files.

Create new project directories at a temporary location, e.g.

jekyll new /tmp/newsite

Commit these files to the SVN repository (I normally check out the repository on my local workstation, copy the directory in /tmp from the server into the working directory on the workstation, add them and commit). Delete the directory in /tmp.

On the server, create the actual website file location by exporting from the SVN via a temporary location:

svn export file:///path/to/repository /tmp/buildfiles
jekyll build --source /tmp/buildfiles /var/www/sitename
rm -Rf /tmp/buildfiles

Configure Apache to serve from /var/www/sitename. In our case we ultimately wanted to serve multiple sites through a reverse proxy, so we used a vhost serving on an alternate port. This can be a handy testing configuration – you don’t have to worry about fiddling with the other website settings. For example, using port 8081:

<VirtualHost *:8081>

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/sitename

</VirtualHost>

(Remember to change ports.conf to listen on the new port!)

Test by pointing a webserver at server:8081

Once that’s all working, set up the post-commit hook script to automatically build the site on a commit. Our current setup is:

#!/bin/sh

# POST-COMMIT HOOK

REPOS="$1"
REV="$2"
REPOS_BASENAME=$(/usr/bin/basename "$REPOS")
TMP_SVN_EXPORT="/tmp/$REPOS_BASENAME"

# These two need configured!
PUBLIC_WWW="/var/www/sitename"
BUILD_EMAIL="your.email@this.address"

"$REPOS"/hooks/mailer.py commit "$REPOS" $REV "$REPOS"/hooks/mailer.conf

LOGVAR=$(/export0/svn_config/jekyll_build.sh "$REPOS" $REV "$TMP_SVN_EXPORT" "$PUBLIC_WWW" 2>&1)

echo "$LOGVAR" | /usr/bin/unix2dos | mail -s "$REPOS_BASENAME build $REV" "$BUILD_EMAIL"

(Note that on Debian you need to install the dos2unix package. Needed as plain text email expects CRLF line terminators as specified in RFC 2822.)

#!/bin/sh

REPOS="$1"
REV="$2"
TMP_SVN_EXPORT="$3"
PUBLIC_WWW="$4"

/usr/bin/svn export --quiet file:///"$REPOS" "$TMP_SVN_EXPORT"
/usr/bin/jekyll build --source "$TMP_SVN_EXPORT" --destination "$PUBLIC_WWW"
rm -Rf "$TMP_SVN_EXPORT"

Note that the build process runs under the Apache user account, so set permissions appropriately. Also, when troubleshooting remember that on Debian 9 the Apache process is configured by default to use a private /tmp directory!

This works for our current needs, although it isn’t optimised. Improvements would be:

  • Unify the setup for the commit email and build email scripts.
  • Build the site in the background (although you’d have to tweak how the logging output works in that case).

Of course, the professionals would use something like a combination of GitLab and Jenkins to automate this stuff properly…

 

Private /tmp directories in Debian 9 Stretch with Apache

In Debian 9 Stretch Apache is configured to use systemd‘s PrivateTmp feature by default. This means that the Apache tmp directory actually lives in /tmp/systemd-private-BIGLONGSTRING--apache2.service-STRING.

So if you are running an SVN server that uses Apache for serving, anything written to /tmp in the hook scripts ends up in the private directory rather than the normal userspace one.

502 Bad Gateway with SVN copy, move or rename behind a reverse proxy

Problem: You have a SVN server sitting behind a reverse web proxy (e.g. for convenient SSL termination purposes). This works for new files, changes etc. but fails when you try to rename something, make a copy or move. The error is:

Unexpected HTTP status 502 'Bad Gateway'

The reason is explained here, but to summarise:

These operations involve sending a http COPY method. This includes a Destination: field in the header, which is not rewritten by Apache’s ProxyPass directives. Thus the destination field starts with https – not http. This confuses mod_dav on the SVN server.

The solution is to change the header. We can do this on Apache (2.2 or higher) using the headers module. This can be done either on the proxy server or the SVN server. As my SVN server is very old (the main reason why it’s behind a proxy) I’ll do this on the proxy server.

Enable the headers module if required. On Debian:

# a2enmod headers

and restart Apache. Then alter your configuration to include:

RequestHeader edit Destination ^https http early

This probably should go before any ProxyPass directives.

Then your config might look like:

RequestHeader edit Destination ^https http early

ProxyPass /svn http://your.real.svn.server/svn
ProxyPassReverse /svn http://your.real.svn.server/svn
<Location /svn/>
  Require all granted
</Location>

(Note this is for a gateway system where other locations can proxy to other application servers.)

Checking out SVN in a new directory and getting a ‘working copy too old’ (or similar) error

Had a situation today where we were trying to check out a SVN repository and kept getting

Check Out: Cleanup with an older 1.7 client before upgrading with this client

both with SmartSVN and the OSX command line svn – into a new clean directory.

The problem turned out to be an old .svn metadata folder in the directory above which should have been deleted when rearranging folders. This seemed not to affect existing working copies below this, but it looks like it did cause problems with creating new working copies. Deleting the rogue .svn directory made things work.

CentOS setup for Subversion Edge

N.B. Gave up on this – taking too much time and effort for something that would require some command line admin anyway – such as the mailer configuration. Will go back to purely standard setup on Debian like existing servers.

Minimum setup

Network

To get network going edit ifcfg-eth0  in /etc/sysconfig/network-scripts/

Need to use vi for this – minimal install doesn’t have anything else.

DEVICE="eth0"
NM_CONTROLLED=no
ONBOOT=yes
HWADDR=6E:89:F7:F2:8B:47
TYPE=Ethernet
BOOTPROTO=none
IPADDR=130.209.45.49
NETMASK=255.255.255.0
GATEWAY=130.209.45.1
DNS1=130.209.4.16
DNS2=130.209.4.18
DNS3=130.209.16.6
DOMAIN=physics.gla.ac.uk
IPV6INIT=no
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

The UUID here was put in by the GUI network setup, as was some of the other stuff. The Minimum required is IPADDR, BOOTPROTO (maybe?), NETMASK, GATEWAY, DNS1(2,3,etc).

To edit file with vi:

vi filename
i (insert mode)
ESC (command mode)
😡 (Exit, saving changes)
:q! (Exit and ignore any changes)

vi cheat sheet at http://www.lagmonster.org/docs/vi.html

Enable network with service network start

Edit /etc/yum/conf to include the line

proxy=http://wwwcache.gla.ac.uk:8080/

then yum update to update system. Then use yum install packagename to install other stuff, like nano, ntp etc.

Install sudo. seems to want it.

Firewall

Install system-config-firewall-tui using yum and run it.

Java

yum install java-1.6.0-openjdk

Set environment variable by creating a file in /etc/profile.d/

export JAVA_HOME=/usr/lib/jvm/jre

(if the install worked properly /usr/lib/jvm/jre should be a symlink to /etc/alternatives/jre which in turn should link to the actual jre, e.g./usr/lib/jvm-exports/jre-1.6.0-openjdk.x86_64)

Python

Python should be installed. Apparently need to install python-ldap though.

User

Create user with:

useradd username -p password

Install

Create directory, e.g.

mkdir /export0/subversion_edge
chown subversion:subversion /export0/subversion_edge/

Put the downloaded file in subversion user’s directory, switch to the user and extract the file

tar -zxvf CollabNetSubversionEdge-3.1.0_linux-x86_64.tar

change back to root (su -) and:

[root@unst ~]# /export0/subversion_edge/csvn/bin/csvn install
Detected RHEL or Fedora:
 Installing the CSVN Console daemon..
Setting RUN_AS_USER to: 'root'. Please edit '../data/conf/csvn.conf' if this needs to be adjusted
Setting JAVA_HOME to: '/usr/lib/jvm/jre'. Please edit '../data/conf/csvn.conf' if this needs to be adjusted.

Edit the csvn.conf file to change the RUN_AS_USER. Can also set the proxy here.

Change back to the subversion user and start the system

/export0/subversion_edge/csvn/bin/csvn start

After a minute or so should be able to access the console at http://servername:3343/csvn or https://servername:4434/csvn/

For troubleshooting this gives you the startup messages:

csvn console

Starting svn server on port 80 or 443

To allow the non-privileged user to bind to ports < 1024 there are some tweaks required. Either:

  1. Give root ownership of the httpd_bind utility to allow it to start the server from init.d, or
  2. edit sudoers to give allow subversion user to start apache

When you set a port < 1024 in the web interface for the svn server it will complain and show the commands required to make it work, for both options. I chose the httpd_bind route.

 chown root:subversion /export0/subversion_edge/csvn/lib/httpd_bind/httpd_bind
 chmod u+s /export0/subversion_edge/csvn/lib/httpd_bind/httpd_bind

https://ctf.open.collab.net/sf/go/artf5095?nav=1&selectedTab=comments

Setting daemons to start

To start the console and the svn/httpd services on boot:

csvn/bin/csvn install
csvn/bin/csvn-httpd install