Configuring a webserver using OpenBSD with httpd(8), relayd(8), and acme-client(1)

Overview

I wanted to move my rss aggregator miniflux.app off Miniflux’s (excellent, but paid) hosting option, and onto my own existing OpenBSD server. Miniflux is somewhat unique, in that it provides a single binary; no docker containers here!

Both that binary miniflux and postgresql-contrib are available via pkg_add(1)

This is the first time I’ve setup a server like this, particularly public-facing; so its taken a fair bit of trial and error and I thought it would be good to jot things down in case this helps others.

My objective was simple.

Have an already owned domain (this one!). Someone heading to the front page of this domain should (hopefully) see posts such as this.
Have miniflux’s web-server be accessible, just as if I was visiting reader.miniflux.app.
Have the whole thing served with TLS - for sensible reasons - and also 3rd party clients require it.
behind a reverse-proxy using relayd(8)

As this is my first time doing this, any suggestions of how to improve my configuration; or helpful alerts to critical errors will be greatly appreciated.

Thank You’s

Before I continue, it is essential that I thank the following.

Michael Lucas - mwl.io: Michael is the sole reason that I got into BSDs. I’d stumbled upon his talk about FreeBSD’s jails and was immediately drawn in. My former employer gave me access to his books, and here I am today. Important for this project is “Lucas, M, W (2017) Relayd and Httpd Mastery (IT Mastery Book 11), Tilted Windmill Press”
Michał Sapka’s straight forward introduction to httpd(8 and relayd(8).
TuM’Fatig’s (slightly) more complicated explanation of “relayd(8) as an application layer gateway”; well beyond my level of comprehension, but there were a few smaller points in the various examples that I found helpful.
Openbsd community in general, particularly r/openbsd.

Frustrations

I’m going to get this out of the way first. This was frustrating. This is not the fault of anyone; except for perhaps my own lack of knowledge. I perhaps took on a bit too much when I dived into this project over the weekend. There are a few things that stood out to me though.

SSL certs are, of course, extremely important, and a modern web browser or other application, is probably going to throw a lot of tantrums if you’ve got something misconfigured. Some applications (remaining nameless), are even so kind as to not give you any sort of error message explaining their tantrum. Miniflux will need to be pointed to certificates - importantly the fullchain.crt, not the standard .crt that you receive from acme-client(1). Further, relayd.conf(5) and httpd.conf(5) can of course support this traffic - but need to be explicitly told to do so.

Miniflux’s binary file looks for environmental variables, it does not read the config file by default. The config file is read by the daemon, only by the daemon. You can force the miniflux binary to read the config file by using miniflux -c /etc/miniflux.conf. This was the first, and longest source of much head-banging & gnashing of teeth.

Miniflux & postgresql (_miniflux & _postgresql) have their own users, and are to be contained. They should be given strong & complicated passwords. Importantly, the miniflux user in postgresql is not _miniflux the daemon user.

Process

Okay, with the nasties out of the way; here’s what I accomplished. All commands are being run as root unless otherwise indicated

Part 1 - Miniflux

Start with installing miniflux and postgresql-contrib via pkg_add(1). Both packages are pretty small downloads, and this shouldn’t take much time at all. I then set a password for both passwd _miniflux & passwd _postgresql. Importantly, as I learnt from Luca Ferrari’s guide, there needs to be ‘data’ database configured before the postgresql daemon (_postgresql) can be run. Once that’s configured, following the instructions given with the miniflux package readme, will create a database for miniflux with a database user & password. Although I’d suggest the following tweaks.

Give the user _miniflux its own folder

mkdir $/path/to/folder # ideally somewhere away from where it can do any harm should it go rogue.
chown _miniflux $path/to/folder

Create the database and assign the (database user) miniflux as it’s owner

su - _postgresql
createuser -P miniflux
createdb -O miniflux miniflux
exit # back to root
psql miniflux -c 'create extension hstore'

This new database can then be pointed to in /etc/miniflux.conf. Assuming a database user named miniflux

DATABASE_URL Postgresql connection parameters                                                                         
(default: user=postgres password=postgres dbname=miniflux2 sslmode=disable) DATABASE_URL=postgresql://miniflux:$passwd@127.0.0.1/miniflux?sslmode=disable

Then, ensuring miniflux has the correct config

su - _miniflux
. /etc/miniflux.conf
miniflux -c /etc/miniflux.conf -debug -migrate # if you don't pass miniflux as a config file, it will load environmental variables
miniflux -create-admin # this can be changed immediately after you first log in.

assuming no errors or issues

rcctl enable miniflux
rcctl start miniflux

Now’s probably a good time for sanity checking, and loading up the LISTEN_ADDR in your browser. If you’re doing this across ssh, don’t forget you can use ssh -L localport:remotehost:remoteport $user@$address to tunnel - I can’t remember where I first learnt this trick, but its insanely useful.

Hopefully you’re greeted with the login page for miniflux. From there you can configure your user. We’re not done with the miniflux config just yet, but its a good starting off point.

Part 2 - `httpd(8)`

I largely followed Michael Lucas’ book here, and took the time to read through most of it. If you don’t have a copy handy - shameless plug - there are a few more ’terse’ sources of information such as the OpenBSD Handbook. To start with, our objective is to setup httpd to recieve traffic on ports 80 & 443, with a directory for the Let’s Encrypt challenge to get our certificates.

  server "www.example.com" {
         listen on $public_ip port 80
         root "/$website dir"
         location "/.well-known/acme-challenge/*" {
       		 root "/acme"
    		 request strip 2
  	}
  }

Then to ensure it’s working correctly httpd -dn, which will print out a nice debug message of any errors - some /other/ pieces of software relevant to this guide could learn a thing or two. If all is well rcctl enable httpd && rcctl start httpd

Our aim now is to get acme-client(1) configured & run to ensure we have certificates to allow https, and clients to connect to miniflux’s API. Note: for those following with Mr. Lucas’ book, root strip 2 in httpd.conf is outdated, and should be replaced with request strip 2

We need to add our domain to /etc/acme-client.conf before we can proceed - from the OpenBSD Handbook.

domain example.com {
         alternative names { www.example.com }
         domain key "/etc/ssl/private/example.com.key"
         domain certificate "/etc/ssl/example.com.crt"
         domain full chain certificate "/etc/ssl/example.com.fullchain.crt"
         }

Make the required directories & then run acme-client(1)

mkdir -p -m 700 /etc/acme
mkdir -p -m 700 /etc/ssl/acme/private
mkdir -p -m 755 /var/www/acme

acme-client $website && rcctl reload httpd

Part 3 - Certs

Hopefully there’ll be a message that indicates the process has been successful. Your server now has a key so its important that we configure httpd(8) again. But first as mentioned above, miniflux & its clients are fussy about how certs work. We’ve already isolated the _miniflux user, and the daemon has its own sandpit to play in, so we’ll shove a copy of the fullchain.crt & private.key into this folder. I’m not sure that this is the most secure method, but I think its better that we don’t give _miniflux access to /etc/ssl It is important that the fullchain.crt is used here, web browsers are smart, miniflux clients aren’t - and they won’t tell you they’re not smart either! grrr…

To ensure each process is reading the same keypair (see relayd section) I’m renaming the $website.fullchain.pem as $website.crt. This seems to be functioning, and keeping web browsers and android apps collectively happy.

mv /etc/ssl/$website.crt /etc/ssl/$website.crt.bak
cp /etc/ssl/$website.fullchain.crt /etc/ssl/$website.crt
cp /etc/ssl/$website.fullchain.pem $miniflux_folder/
cp /etc/ssl/private/$website.key $miniflux_folder/
chown _miniflux $miniflux_folder/$website.key

rcctl reload httpd
rcctl reload relayd

You also need to tell miniflux.conf where the keys are. It can (hopefully) only access its own folder.

# CERT_FILE Path to SSL certificate (default: None)                                                                     
CERT_FILE=$miniflux/$website.fullchain.pem

# KEY_FILE Path to SSL private key (default: None)                                                                      
KEY_FILE=$miniflux/$website.key

As certs only last 90 days, others’ articles have suggested a script which can be run via crontab. Put this somewhere, ensure it’s executable and pop it in crontab -e

Part 4 - Crontab

WIP - crontab script (as root)

  #!/bin/sh
  acme-client $website && \
  mv /etc/ssl/$website.crt /etc/ssl/$website.crt.bak && \
  cp /etc/ssl/$website.fullchain.pem /etc/ssl/$website.crt && \
  cp /etc/ssl/$website.fullchain.pem $miniflux_folder/ && \
  cp /etc/ssl/private/$website.key $miniflux_folder/ && \
  chown _miniflux $miniflux_folder/$website.key && \
  rcctl reload httpd && \
  rcctl reload relayd && \
  rcctl restart miniflux

Configuring a webserver using OpenBSD with httpd(8), relayd(8), and acme-client(1)#

Overview#

Thank You’s#

Frustrations#

Process#

Part 1 - Miniflux#

Part 2 - httpd(8)#

Part 3 - Certs#

Part 4 - Crontab#