Monitoring Docker Containers with Grafana Alloy and cAdvisor

July 20, 2026 · By Justyn Larry

TL;DR: Host-level metrics tell you a server is healthy, but they don’t tell you if there’s a problem at the container level. cAdvisor reads container usage directly from the kernel’s cgroup data, and Alloy scrapes it over localhost and folds those samples into the same remote_write stream already carrying host metrics and logs. Because cAdvisor sees everything in the cgroup tree, it needs to be filtered to keep noisy metrics out.

// CONTAINER MONITORING

A few recent posts talked about choosing between Prometheus Agent mode and Alloy and the process of migrating from node_exporter to Alloy. Both are concerned with the agent that ships host metrics and logs upstream. The natural follow-up is monitoring the Docker containers running on that same host. My own stack runs off several Docker Compose files, around 18 containers, and that’s just on one server. Watching host-level CPU and memory shows the server is fine, no issues, but if there’s a runaway container, those metrics won’t tell you which container is eating all the memory.

cAdvisor is the standard solution to that problem. It’s an open-source project out of Google that reads container resource usage straight from the cgroup (Control Group, the underlying Linux kernel feature that tracks and enforces resource limits) and namespace data the kernel already tracks, and it requires no instrumentation inside the container itself. This post covers how it fits into a push-based agent architecture without opening anything up.

// DEPLOYMENT

cAdvisor runs as its own container, separate from Alloy, with a fairly wide set of read-only mounts so it can see the host’s cgroup and Docker state:

docker run \
  --name cadvisor \
  --privileged \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=<port>:8080 \
  gcr.io/cadvisor/cadvisor:latest

At a glance, the --privileged flag combined with read access to the root filesystem can look alarming, but every volume it needs is mounted :ro (read-only). cAdvisor is reading host state, not writing to it, and the privilege is there so it can see cgroup and device metadata it otherwise couldn’t from inside a container.

You should not hardcode port 8080. It’s one of the most commonly claimed ports on any dev box or small server, used by everything from other proxies to ad hoc web servers, and it is likely to create conflicts down the road. The bootstrap script I use to install Alloy checks a short list of candidate ports starting at 9338, and falls back to 9338 if none of them are free. That port is then reported back to the config server so the scrape target gets generated with the correct value. It’s best to pick a default port ahead of time and prepare a fallback as well, otherwise you risk hitting a conflict down the road on a service you’d forgotten about.

// WIRING IT INTO ALLOY

This is the part that actually connects to the last two posts. cAdvisor exposes a Prometheus-format endpoint, but I don’t point my central Prometheus at it directly. Alloy scrapes cAdvisor over localhost, on the same host, and folds those samples into the same remote_write stream carrying node metrics and logs. From the backend’s point of view, container metrics show up exactly the way host metrics do: pushed in, already labeled, no inbound connection to the client ever required.

This matters more for containers than it does for host metrics. A client running Docker on a residential connection, behind CGNAT, with an IP that changes frequently, is a completely normal setup. It’s also the reason I prefer Alloy over node_exporter: tracking IP addresses for every server and workstation I want to monitor, then updating them on the monitoring server whenever they changed, was creating too much extra work. A local scrape and remote push solves that problem the same way it solved host metrics, and once the pattern exists in your agent it costs nothing to extend it to a new local exporter. The payoff of consolidating everything into one agent is that cAdvisor becomes a second scrape target in an existing pipeline, not a second monitoring system.

// THE CARDINALITY TRAP

One of the things that usually gets overlooked in the getting-started docs is that cAdvisor doesn’t only report metrics for your named containers. It reports metrics for every cgroup it can see. That includes the container’s own wrapper cgroup, kernel-level slices, and other entries with no Docker-assigned name at all.

Left unfiltered, that means duplicate and junk series for infrastructure that isn’t a container you care about, which multiplies your active series count for no useful signal. The simplest fix is to consistently filter cAdvisor metrics with a PromQL label matcher, either in your dashboard queries or by dropping unwanted series at scrape time:

container_cpu_usage_seconds_total{name!=""}

Using this label matcher, you’re only keeping series where the container name label is non-empty. Every dashboard panel and every alert rule built on cAdvisor metrics in my stack carries it. Skip it in a dashboard and you’ll see phantom containers with no name in a dropdown, or a sum() that’s double what docker stats shows on the box itself.

// THREE BASIC ALERTS

Three cAdvisor-based alerts have caught more real problems for me than anything else in this category, and none of them are complicated.

Restart loops, usually the first sign something is actually broken rather than just under load:

changes(container_start_time_seconds{name!=""}[1h]) > 3
and
rate(container_start_time_seconds{name!=""}[15m]) > 0

Sustained high CPU relative to the host’s total capacity, not just one core:

(
  rate(container_cpu_usage_seconds_total{name!="", name!="POD"}[5m])
  / on(instance, tenant) group_left() count by (instance, tenant) (
      node_cpu_seconds_total{mode="idle"}
    )
) * 100 > 80

And memory consumption relative to total host memory, not just the container’s own limit, since a memory leak in one container can starve everything else on the server before the container’s own limit ever trips:

(container_memory_usage_bytes{name!="", name!="POD"}
  / on(instance, tenant) group_left node_memory_MemTotal_bytes) * 100 > 85

That last one is a little tricky if you haven’t written a cross-metric join before. The group_left joins a container-level metric against a host-level metric, so you get container memory as a percentage of total host memory, not an absolute byte count that’s meaningless without context. It’s a good pattern to keep in mind any time you need to relate a fine-grained metric back to something coarser.

// WHERE THIS FITS

Docker monitoring is included on every paid tier, Starter, Pro, and Business, with nothing extra to unlock between them. The only gate is free versus paid. bootstrap.sh detects Docker during onboarding, it can be toggled per server in the user interface, and it’s synced into the client’s Alloy config automatically once enabled. The architecture underneath it isn’t Irin-specific. Any push-based agent that already handles host metrics can absorb a local exporter like cAdvisor the same way. The only caveat is that you need the label matcher that keeps polluted container metrics out.

If you’re running cAdvisor already, I’d be curious whether you’ve hit the unnamed-cgroup problem, or found a different filter that catches cases mine doesn’t.