Self-Hosted Linux Monitoring: A Practical Guide
By Justyn Larry
TL;DR
You do not need Kubernetes, a SaaS contract, or a platform team to monitor a fleet of Linux servers well. You need one agent per host, a place to store metrics and logs, and a small number of decisions made deliberately rather than by default. This guide walks through those decisions in the order they actually come up: which agent to run, whether it should push or pull, which signals to collect, what it costs compared to a hosted service, and what changes when you run it for real across more than a couple of machines. Each section links to a deeper write-up on that specific choice.
Who this is for
If you run somewhere between three and fifty Linux servers, know what Prometheus is, and have either tried self-hosting a monitoring stack and abandoned it or are flying blind and know it, this is written for you. It is opinionated, because “it depends” is not useful when you are trying to get something running this week. If a choice genuinely depends on your situation, the guide makes it clear and provides a question.
A note on where this comes from: everything here is drawn from running this exact stack in production, not from a vendor’s reference architecture. When there’s a tradeoff, it is presented as one.
// THE FIVE DECISIONS
1. Choosing an agent
The agent is the one piece of software that runs on every server you monitor, so getting it right is important. The modern field has narrowed to a few real options, and the consolidation happening across the ecosystem (Grafana Agent and Promtail both reached end of life in 2025 and 2026) has made the decision clearer than it was even a year ago.
The short version: if you only need metrics and your team already speaks Prometheus configuration fluently, a lightweight metrics-only forwarder is hard to beat. If you need to start monitoring logs or traces, a unified agent that handles all three signals in one service earns its slightly steeper learning curve.
Read next:
- Prometheus Agent Mode vs Grafana Alloy — the framework for choosing between a metrics-only forwarder and a unified pipeline.
- Why shifting from node_exporter to Alloy changed our fleet management, security posture, and operational overhead.
- Migrating from node_exporter to Alloy, one server at a time, with no metrics gap.
// PUSH VS PULL ARCHITECTURE
2. Push vs pull architecture
This is the decision most people make without realizing they made it, and it determines what your firewall has to allow, whether you can monitor machines behind NAT, and how your down-detection works.
In the classic Prometheus model, the server reaches out and scrapes each target on a schedule. It is simple and well understood, but it requires network reachability to every host and an open port on each one. It also requires constant maintenance if servers are added or removed. In a push model, the agent on each host initiates an outbound connection and pushes its data upstream, which means nothing listens for inbound connections and machines behind CGNAT or dynamic IPs just work. The tradeoff is that some things taken for granted in the pull world, most notably the built-in up metric, stop existing and have to be rebuilt.
Read next:
- Migrating from node_exporter to Alloy — the same post covers the pull-to-push posture shift in detail.
- Migrating from Promtail to Alloy for log collection, the log-shipping half of the same architecture.
// METRICS, LOGS, AND TRACES
3. Metrics, logs, and traces
These are the three signals of observability, and the common mistake is treating them as one purchase decision when they answer completely different questions. Metrics tell you that something is wrong. Logs tell you what the system said while it went wrong. Traces tell you where, across a request’s path, the time actually went.
Most teams need metrics first, add logs second when metrics alone stop being enough to explain an incident, and start thinking about traces only once they are debugging across multiple services. Knowing which signal answers which question keeps you from over-building on day one or discovering a blind spot mid-incident.
Read next:
- Metrics tell you something broke; tracing tells you what, where, and why.
- Uptime Kuma tells you that a service broke, not why, the line between up/down checks and internal metrics.
- The only five PromQL queries you really need to catch most real problems on a single host.
// COST AND BUILD-VS-BUY
4. Cost and build-vs-buy
When weighing this decision, it’s important not to frame it as “self-hosted is cheaper.” It’s better to think of it as “self-hosted trades money for time, and whether that trade is worth it depends on what your time is worth and how much of it you have.” A hosted service bills per host or per metric and the number climbs as you grow; a self-hosted stack has a flat infrastructure cost and a variable cost in your own attention.
The trap in hosted pricing is rarely the sticker number. It is the metering surprises, the per-host multipliers, and the custom-metric charges that turn a predictable line item into a variable one. Run the math for your fleet size before deciding, because the answer can change depending on where you sit.
Read next:
- Datadog vs open-source monitoring: the real per-host cost math and an honest comparison for small teams.
- Monitoring 20 Linux VMs with Prometheus, no Kubernetes required.
- When you bring your data home, who keeps an eye on it? Observability as the capability teams lose in a cloud repatriation.
// RUNNING IT FOR REAL
5. Running it for real
Everything above is about standing the stack up. This is about what changes once it is carrying real load across more than a handful of machines: isolating one set of servers from another, keeping the monitoring stack itself honest and reliable, and the architectural decisions that are cheap to make early and expensive to retrofit.
Read next:
- How three-layer tenant isolation works without dedicated infrastructure per client.
- How I built multi-tenant observability on a two-server homelab, and what I’d do differently.
- The origin story: building a managed observability SaaS from a homelab, and still not being able to explain what was happening.
// WHERE IRIN FITS
This guide is deliberately tool-focused rather than product-focused, because the decisions above are yours to make whether or not you ever use a managed service. If you get to the end of it and conclude that self-hosting is the right call for your team, that is a good outcome, and the posts linked here will help you do it well.
If you get to the end and conclude that you would rather not spend your attention on running the monitoring stack itself, that is where Irin comes in. Irin is a visibility layer: advisory, minimal footprint, no remote access, outbound-only. It provides you with the instruments and leaves the decisions to you. It is a place to see your systems, not a dependency inside them. That is the whole idea, and it is the same theme running through every post linked above.