I’ve written quite a bit about my homelab setup, and how almost all of the services I run are in Docker containers. I’ve been able to maintain pretty good uptime, but ran into a problem: if something went down or stopped working, how would I know? In general, I wouldn’t, and that made me a little uncomfortable.

Here’s how I solved that.

Healthchecks

The first thing I did was to implement healthchecks for every container. While checking if a container is running or not is a pretty good indicator if a service is running, it’s better to actually check if the service is healthy and responding as you’d expect. It’s completely possible for a container to be running, but the service is not.

Some Docker image authors supply a healthcheck as part of their Docker Compose examples, but I found most do not. So, I turned to Google’s Gemini to help me craft healthchecks for nearly everything running in my homelab.

My initial prompts were similar to this:

Please help me write a healthcheck for the Docker image in amir20/dozzle

From there, it gave me some YAML that I could include in my Docker Compose file for Dozzle:

  # --- DOZZLE (Real-time Docker Log Viewer) ---
  dozzle:
    image: amir20/dozzle:latest
    container_name: dozzle
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - 8080:8080
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "/dozzle", "healthcheck"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

and a way to test the healthcheck to make sure it actually works:

docker exec dozzle /dozzle healthcheck && echo "Healthy"

Great, I copied and pasted that into my Docker Compose file, rebuilt the container, and was able to see that instead of the container simply running, it’d show as healthy. I was checking within Dozzle, but you can also use docker ps to see a container’s health status.

Many times, it’d give me a command that didn’t work, so I tested each to make sure they’d function as intended. Something else to be aware of: some healthchecks are better than others. I’d often ask Gemini to improve its check, because it would generate log spam, either in access logs (if it’s a web service) or in the container logs. Given that these checks run every few seconds, that can add up quickly, chewing up RAM or disk space.

Uptime Kuma

Next, I needed a way to be notified if a container was down or unhealthy. I’ve used Uptime Kuma for a while now, and I really like it. It’s very lightweight, but also very flexible.

In my first iteration, I ran Uptime Kuma as just another container. Hopefully, you’re seeing the obvious issue here: if my home server goes down, so does Uptime Kuma. There’s nothing to send notifications.

I moved this to a separate host, which somewhat solved the problem. I grabbed a spare Intel Mac mini, put the latest version of Ubuntu LTS on there, installed Docker, and spun up exactly one container for Uptime Kuma.

services:
  uptime-kuma:
    container_name: uptime-kuma
    image: louislam/uptime-kuma:2
    # https://hub.docker.com/r/louislam/uptime-kuma
    env_file:
      - /data/docker/uptime-kuma/env_config.txt
      - /data/docker/uptime-kuma/env_secrets.txt
    label_file: /data/docker/uptime-kuma/labels.txt
    volumes:
      - /data/docker/uptime-kuma/data:/app/data:rw
    network_mode: host
    restart: unless-stopped

Problem solved, right?

Nope, because when my home internet went down, I was back to square one. To solve this, I needed to find a way for my Uptime Kuma server to securely talk to my home server, but over the internet. This is why I love Tailscale.

First, I set up LinuxServer.io’s docker-socket-proxy container on my home server. I exposed a random port in the Docker Compose file and set the whole thing to read-only access.

services:
  docker-socket-proxy:
    container_name: docker-socket-proxy
    image: lscr.io/linuxserver/socket-proxy:latest
    # https://github.com/linuxserver/docker-socket-proxy/pkgs/container/socket-proxy
    env_file:
      - /mnt/user/appdata/docker-socket-proxy/env_config.txt
      - /mnt/user/appdata/docker-socket-proxy/env_secrets.txt
    label_file: /mnt/user/appdata/docker-socket-proxy/labels.txt
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - "23750:2375"
    read_only: true
    tmpfs:
      - /run
    healthcheck:
      test: ["CMD-SHELL", "wget --spider -q http://localhost:2375/version || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    restart: unless-stopped

My env_config.txt file looks like this:

CONTAINERS=1
EVENTS=1
INFO=1
NETWORKS=1
POST=0
SERVICES=1
TASKS=1
VERSION=1

In Uptime Kuma, I added my home server in Settings > Docker Hosts. I set it as TCP / HTTP, and used my home server’s Tailscale IP:

tcp://100.71.177.26:23750

I tested it, and it worked. I could move this Mac mini to another network and it’d continue to monitor my containers.

Alright, so at this point I could add every single container to Uptime Kuma, and it’ll monitor the health state of each one. That’s a lot of work, though. A lot of clicking through Uptime Kuma. Plus, if I spin up a new service, that’s going to require me to remember to click through everything all over again. Can I make this easier for myself?

AutoKuma

Yep! This is where another project named AutoKuma comes in. Using Docker labels, I was able to populate Uptime Kuma using text files.

Here’s my Docker Compose file for AutoKuma:

services:
  autokuma:
    container_name: autokuma
    image: ghcr.io/bigboot/autokuma:2.1.0-rc.2
    # https://github.com/BigBoot/AutoKuma/pkgs/container/autokuma
    env_file:
      - /mnt/user/appdata/autokuma/env_config.txt
      - /mnt/user/appdata/autokuma/env_secrets.txt
    label_file: /mnt/user/appdata/autokuma/labels.txt
    volumes:
      - /mnt/user/appdata/autokuma/data/database:/data
      - /mnt/user/appdata/autokuma/data/autokuma.toml:/autokuma.toml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    network_mode: host
    restart: unless-stopped
    depends_on:
      - docker-socket-proxy

My env_config.txt file contains only this:

AUTOKUMA__KUMA__URL=https://uptimekuma.tails-scales.ts.net:3001

and my labels.txt file contains this:

kuma.meatball.docker_host.name=myhomeserver
kuma.meatball.docker_host.connection_type=tcp
kuma.meatball.docker_host.host=tcp://100.71.177.26:23750

kuma.Databases.group.name=Databases
kuma.Downloaders.group.name=Downloaders
kuma.Libraries.group.name=Libraries
kuma.Other.group.name=Other
kuma.Server.group.name=Server

kuma.autokuma.docker.name=autokuma
kuma.autokuma.docker.docker_container=autokuma
kuma.autokuma.docker.docker_host_name=myhomeserver
kuma.autokuma.docker.parent_name=Server

I’ve broken this down into three sections:

  • The first section automates the Docker Host we manually configured above – since AutoKuma won’t reuse hosts you’ve created in the UI, just delete yours and recreate it via AutoKuma instead.
  • The second section is for creating groups in Uptime Kuma, so you can categorize what you’re monitoring. I found it easiest to put this in AutoKuma’s labels.
  • The third section is for monitoring AutoKuma itself via Uptime Kuma (because why not? 😄 ).

With that running, you can now add a handful of labels to any container and AutoKuma will use those to populate Uptime Kuma. Here’s an example with Dozzle:

kuma.dozzle.docker.name=dozzle
kuma.dozzle.docker.docker_container=dozzle
kuma.dozzle.docker.docker_host_name=myhomeserver
kuma.dozzle.docker.parent_name=Server

After rebuilding the Dozzle container, AutoKuma will pick up these labels and add dozzle to Uptime Kuma within seconds.

From there, you can add downtime notifications, since you probably don’t want to constantly check Uptime Kuma to make sure things are running properly. I find Pushover to be great for this. Just configure it manually in Uptime Kuma and note the ID. Create a file called autokuma.toml and populate it with this:

default_settings = '''
*.notification_id_list: { "1": true }
'''

(change 1 to the ID of your Pushover notification configuration)

We’re done, right? But wait, what happens if my Uptime Kuma server goes down? How can I monitor for that?

Healthchecks.io

Somewhat confusingly, this is not the same healthchecks I detailed at the beginning of this blog post. I’m using a separate tool called Healthchecks.io to monitor my Uptime Kuma server.

Healthchecks.io works a little differently: it generates a unique URL for each “thing” you want to monitor: it can be a server’s uptime, the completion of a script, practically anything.

I spun up Healthchecks.io on my home server:

services:
  healthchecks:
    container_name: healthchecks
    image: healthchecks/healthchecks
    # https://hub.docker.com/r/healthchecks/healthchecks
    env_file:
      - /mnt/user/appdata/healthchecks/env_config.txt
      - /mnt/user/appdata/healthchecks/env_secrets.txt
    label_file: /mnt/user/appdata/healthchecks/labels.txt
    volumes:
      - /mnt/user/appdata/healthchecks/data:/data:rw
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped

I’m not going to go too deep into setting up Healthchecks.io, since it’s a bit more involved, but be sure to read the documentation on how to set it up. It’s free if you’re running it yourself. I’m using the sqlite database just to keep things as simple as possible.

From there, I set cron on my Uptime Kuma server to ping Healthchecks.io running on my home server every 5 minutes (using that Tailscale IP). If the ping doesn’t arrive, Healthchecks.io sends me a push notification via Pushover.

Of course, I also have Uptime Kuma monitoring Healthchecks.io:

kuma.healthchecks.docker.name=healthchecks
kuma.healthchecks.docker.docker_container=healthchecks
kuma.healthchecks.docker.docker_host_name=myhomeserver
kuma.healthchecks.docker.parent_name=Server

All in all, it looks like all of this monitoring consumes anywhere from 8 to 22 GB of bandwidth each day. Maybe reconsider this project if you have bandwidth caps. 😉

If you’re at all interested in monitoring your home server, I hope this helps!