A recent effort of mine has to been to consolidate the various services and applications I’ve been running across my VPSs and homelab. The last piece of the puzzle was moving jellyfin off bare metal and into a container, a very straightforward process; but part of this transition was to also no longer use this domain as the reverse proxy for any service that I’m running.
So simple, right? At least I thought so. Once I got the container up and running on my homelab, I setup the reverse proxy on another owned domain. As per usual I spent a few minutes troubleshooting things because I forgot to open the port on the server’s firewall, but my own errors aside, everything was working. So I did the last step, cleaning up /etc/relayd.conf.
relayd(8) is OpenBSD’s relay daemon, which can for example, act as a reverse proxy; redirecting incoming connections to other resources, such as my home server. It’s built in on OpenBSD and I’ve been a happy user for a while now. Like many OpenBSD tools, its syntax is straight-forward and located in a configuration file (defaulting to /etc/relayd.conf - see relayd.conf(5)).
So I set about cleaning up /etc/relayd.conf removing references to the various services that vaettiheim.xyz no longer acts as a host or reverse proxy for. This just required deleting a couple of lines in that file. Yet for some reason, removing those relevant lines broke something in relayd, and thus this website no longer loaded. I tried manually readding the deleted lines, only to find that this didn’t solve the issue either. Panic.
I didn’t have the time for a deep dive into the logs for troubleshooting, but thankfully I keep daily backups using restic backup tool - an excellent tool that I’ll eventually write a post about. It was a simple task of restoring the correct file from the backup which (thankfully!) solved whatever the issue was. Restic’s excellent documentation covers the process nicely, and my most recent backup was only a couple of hours old.
This is the first time I’ve needed to restore a backup of a critical file such as this and I’m glad I took the time to setup and configure backups. I’ve read many times other people finding themselves in an unrecoverable situation and I’m really glad that this wasn’t me. I’ve also got snapshots enabled on this server’s vps host, which would be another point of recovery. I think having a couple of layers of redundency, and a ‘defence-in-depth’ approach suits.
Now to figure out why relayd broke.