Pokhara, NepalUTC+05:45 NPT
Open to conversations

Operations

Reading Linux logs without panicking

journalctl, grep, and an order of operations for the first five minutes of an incident.

4 min read

The first five minutes of an incident are mostly about narrowing the search space. The logs are already there. The problem is that there are too many of them.

Start with time

Do not read logs from the beginning. Anchor on when the problem started, then work outward.

journalctl --since "10 min ago" --no-pager
journalctl -u my-service --since "2026-06-11 09:30" --until "09:45"

If you do not know when it started, ask the person who noticed. "It broke around lunch" is still a timestamp.

Then narrow by unit

A whole-system journal is noisy. Move to the service that owns the failure, then to its dependencies.

journalctl -u api.service -p warning --since "1 hour ago"
systemctl status api.service

-p warning removes the informational chatter. If the service is not logging at all, that is itself a signal: it may not have started, or it may be writing somewhere else.

Look for the first error, not the loudest

The last error in a log is often a consequence. The first one is usually the cause. Read upward from the failure and look for the transition from normal to abnormal.

Common patterns worth recognizing:

  • connection refused — nothing is listening, or the wrong host/port.
  • no space left on device — check df -h and inode usage.
  • permission denied — often a recent ownership or mount change.
  • context deadline exceeded — the dependency is slow or unreachable.

Write it down

After the incident, add the queries that worked to the runbook. Not the whole investigation — just the three or four commands that shortened the path. The next person should not have to rediscover them under pressure.

The goal is not to read more logs. It is to read fewer, better ones.