Homelab

Don't be fooled, having a home server is really just hundreds of hours of badblocks.

I tried to fit as many buzzwords into this stack as I could: rootless Podman container orchestration with a ZFS array and SnapRAID JBOD, behind an OPNsense firewall and a Traefik ingress with OAuth, deployed with Ansible to multiple Arch Linux hosts, on a WireGuard network. 🏅

This infrastructure as code is written for me because I'm forgetful. But perhaps it'll help you develop your own server architecture.

Setup

Run setup to create local configuration files and install requirements.yml from ansible-galaxy.

make setup

Then, edit inventory.yml with the target vars and secrets. See example.inventory.yml for what that looks like.

Deploy

If you're unfamiliar with Ansible, it's absolutely worth the effort to learn the mechanics and employ it in your own homelab.

Playbooks

See Working with playbooks and ansible-playbook

Hosts setup and upgrade

• Upgrade pacman and apt cache, packages and the apt distribution.
• Deploy rootless containers in an orchestration behind Traefik's reverse proxy.
• Setup SnapRAID for JBOD disk parity and configure cron to run a snapraid-runner script to sync parity and periodically check the data for errors.

Container composition

Rootless podman support for container images and deployment within an ansible role.

Hardware

Homelab

• custom NAS build

Resources

• DataHoarder Wiki: Hardware
• 2020 DIY NAS
• 2020 Economy DIY NAS
• Home server case recommendations

Memory

• Why use ECC (discussion)

• If you love your data, use ECC RAM.

• Error rates increase rapidly with rising altitude.

Storage

Controller

I used a widely-known and inexpensive method to add additional SATA storage via a Host Bus Adapter (HBA). I purchased a Dell Perc H310 a long while back. Mine did come from overseas, but it turned out to be legit. This video shows how it can be flashed to an LSI 9211-8i IT (see also 1, 2).

Here are other recommended controllers.

2.5" drive stackers

These printable stackers are great for stacking SSDs in a homelab.

Home automation

Zigbee/Z-Wave

• USB Zigbee/Z-Wave receiver and upgrade Zigbee firmware for compatibility with Home Assistant (notice this issue)

Software

Linux

Since I use Arch Linux as my daily driver, it's convenient for me to also use it to run my servers. Debian is a worthy alternative, however I find the stability of Arch to be sufficient for home use. See my linux notes and automation.

Podman

How do user IDs and rootless containers work?

SSH

Here's a nice convenience for setting up authorized_keys stored on Github or Gitlab:

curl https://github.com/<username>.keys -o authorized_keys

You could pipe the output to sed to only grab a specific line sed '4!d'.

Check disks

Here's a handy script to automatically test disks with badblocks and SMART: Spearfoot/disk-burnin-and-testing.

Testing disks takes a long time for larger drives, but it's worth it to be thorough before determining whether to make a return. This is a destructive test, so it's probably best to use /dev/disk/by-id to be certain you're targeting the correct drive.

Use tune2fs -l <partition> to identify the block size.

sudo badblocks -wsv -b 4096 /dev/sda > sda_badblocks.txt

Here's some additional advice from /r/DataHoarders.

JBOD

MergerFS is a union filesystem for pooling drives together. It's a great pair with SnapRAID. An alternative is SnapRAID-BTRFS.

mkdir /mnt/disk{1,2,3,4}
mkdir /mnt/parity1 # adjust this command based on your parity setup
mkdir /mnt/storage # this will be the main mergerfs mount point (a collection of your drives)

Mount drives to these folders, then add /etc/fstab entries by ID.

ls /dev/disk/by-id

You must also include an entry for the MergerFS union, such as:

/mnt/disk* /mnt/storage fuse.mergerfs allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs,fsname=mergerfs,minfreespace=10G 0 0

See also perfectmediaserver: MergerFS

Remember, for data that's irreplaceable RAID is not a backup.

mkinitcpio

Be sure to add zfs and resume

HOOKS=(base udev autodetect modconf block filesystems keyboard zfs resume fsck)

Then, regenerate the image.

ZFS

Install zfs-dkms and zfs-utils, and be sure to have linux-headers installed for dkms to work.

OS Installation

Use Ventoy to bundle bootable ISO and IMG images on a single USB.

Media

For a media server, it's a good idea to understand digital video.

HTTP

Traefik's reverse proxy directs HTTP traffic to the appropriate container based on the service declared by these frontend configurations.

Middlewares configure Traefik in different ways, including security headers. Scan your site to confirm good practice with some headers.