ANSI control codes in Jekyll output breaking emails

Also see Publishing websites with Jekyll, Apache and SVN

If you send console output via email (like, say the output of jekyll build as part of a SVN post-commit hook script) if there are ANSI control characters in the string (e.g. colour codes) this can break things. In this case the mail command (Debian 9 default exim) was only sending text up to the first ANSI code, which meant that the jekyll build error messages (which are yellow and red) were missing.

To fix this pipe the text through ansi2txt (comes with the colorized-logs package in Debian and Ubuntu). This strips out all ANSI control codes making the string email safe.

(After this I pipe it through unix2dos to convert to CRLF line endings, as this appears to be the standard for email. On Debian this comes with the dos2unix package.)

The last line in the hook script then becomes

echo "$LOGVAR" | /usr/bin/ansi2txt | /usr/bin/unix2dos | mail -s "$REPOS_BASENAME build $REV" "$BUILD_EMAIL"

Publishing websites with Jekyll, Apache and SVN

Now I’ve got this working to some extent here are some notes about setting up Jekyll with SVN and Apache:

Server – Debian 9 Stretch, normal command-line only install. Set up system to use email server (campus smarthost in our case).

Install SVN and Apache and set up accordingly.

Install Jekyll:

apt install jekyll

Create an SVN repository for the site files.

Create new project directories at a temporary location, e.g.

jekyll new /tmp/newsite

Commit these files to the SVN repository (I normally check out the repository on my local workstation, copy the directory in /tmp from the server into the working directory on the workstation, add them and commit). Delete the directory in /tmp.

On the server, create the actual website file location by exporting from the SVN via a temporary location:

svn export file:///path/to/repository /tmp/buildfiles
jekyll build --source /tmp/buildfiles /var/www/sitename
rm -Rf /tmp/buildfiles

Configure Apache to serve from /var/www/sitename. In our case we ultimately wanted to serve multiple sites through a reverse proxy, so we used a vhost serving on an alternate port. This can be a handy testing configuration – you don’t have to worry about fiddling with the other website settings. For example, using port 8081:

<VirtualHost *:8081>

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/sitename

</VirtualHost>

(Remember to change ports.conf to listen on the new port!)

Test by pointing a webserver at server:8081

Once that’s all working, set up the post-commit hook script to automatically build the site on a commit. Our current setup is:

#!/bin/sh

# POST-COMMIT HOOK

REPOS="$1"
REV="$2"
REPOS_BASENAME=$(/usr/bin/basename "$REPOS")
TMP_SVN_EXPORT="/tmp/$REPOS_BASENAME"

# These two need configured!
PUBLIC_WWW="/var/www/sitename"
BUILD_EMAIL="your.email@this.address"

"$REPOS"/hooks/mailer.py commit "$REPOS" $REV "$REPOS"/hooks/mailer.conf

LOGVAR=$(/export0/svn_config/jekyll_build.sh "$REPOS" $REV "$TMP_SVN_EXPORT" "$PUBLIC_WWW" 2>&1)

echo "$LOGVAR" | /usr/bin/unix2dos | mail -s "$REPOS_BASENAME build $REV" "$BUILD_EMAIL"

(Note that on Debian you need to install the dos2unix package. Needed as plain text email expects CRLF line terminators as specified in RFC 2822.)

#!/bin/sh

REPOS="$1"
REV="$2"
TMP_SVN_EXPORT="$3"
PUBLIC_WWW="$4"

/usr/bin/svn export --quiet file:///"$REPOS" "$TMP_SVN_EXPORT"
/usr/bin/jekyll build --source "$TMP_SVN_EXPORT" --destination "$PUBLIC_WWW"
rm -Rf "$TMP_SVN_EXPORT"

Note that the build process runs under the Apache user account, so set permissions appropriately. Also, when troubleshooting remember that on Debian 9 the Apache process is configured by default to use a private /tmp directory!

This works for our current needs, although it isn’t optimised. Improvements would be:

  • Unify the setup for the commit email and build email scripts.
  • Build the site in the background (although you’d have to tweak how the logging output works in that case).

Of course, the professionals would use something like a combination of GitLab and Jenkins to automate this stuff properly…

 

Installing Jekyll on Debian 8 (Jessie)

Install ruby-full:

sudo apt-get install ruby-full

Install ruby-ffi

apt-get install ruby-ffi

Install Jekyll

gem install --http-proxy http://wwwcache.gla.ac.uk:8080 jekyll

install node.js

apt-get install nodejs

Should then work for normal user.

Quick test – (from https://www.digitalocean.com/community/tutorials/how-to-deploy-jekyll-blogs-with-git)

jekyll new awesomeblog

This will create an awesomeblog directory containing the configuration files, posts directory and other required bits.

cd awesomeblog
jekyll serve

Jekyll will build your blog, and after a few seconds you should be able to visit http://host:4000 in your browser. Or skip the serve bit and check in the _site subdirectory.

To get git working with this create the git repository, go into the hooks subdirectory and create a post-receive script like this (again from https://www.digitalocean.com/community/tutorials/how-to-deploy-jekyll-blogs-with-git)

#!/bin/bash -l
GIT_REPO=$HOME/repos/awesomeblog.git
TMP_GIT_CLONE=$HOME/tmp/git/awesomeblog
PUBLIC_WWW=/var/www/awesomeblog

git clone $GIT_REPO $TMP_GIT_CLONE
jekyll build --source $TMP_GIT_CLONE --destination $PUBLIC_WWW
rm -Rf $TMP_GIT_CLONE
exit

or /var/www/html/awesomeblog for apache on Debian 8.