Uptime, measured

Start with a question

My first dashboard was a wall of graphs because graphs looked like progress. Most of them never changed a decision. Now I start with a few questions and only keep the panels that answer one:

• Are the services reachable from outside my network?
• Is the host running out of disk or memory?
• Did the nightly backup finish?
• Is anything restarting more often than it should?

If a panel does not help answer one of those, it does not stay.

What I run

Prometheus scrapes node_exporter for host metrics and cAdvisor for container metrics. A blackbox exporter checks the public endpoints, and Uptime Kuma gives me a second, independent view from outside the host. Grafana holds one dashboard, split into host, containers, and endpoints.

I use the same labels across all of them so a service has one name everywhere. That small consistency matters more than it sounds: it is the difference between a dashboard you can read and one you have to decode.

Alerts I kept

The first set of alerts was too broad. High CPU and a single container restart were not things I could act on, and after a week I started ignoring them.

What I keep now:

groups:
  - name: homelab
    rules:
      - alert: EndpointDown
        expr: probe_success{job="blackbox"} == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} has been unreachable for 5 minutes"

      - alert: DiskAlmostFull
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.15
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Root filesystem on {{ $labels.instance }} is below 15% free"

Both alerts describe a state I can do something about. A restart count is a clue, not an instruction.

What is still missing

There is no dead man's switch yet. If Prometheus stops running, the alert rules stop with it and I would not know. The next piece of work is an external check that expects a regular heartbeat and complains when it does not arrive.

Until that exists, Uptime Kuma is doing part of that job by watching from a different machine.