Skip to content

ADR-0012: Grafana Alloy for In-Cluster Container Log Collection

Date: 2026-07-04 Status: accepted Deciders: Tom Morelly (ClearRoute platform team)

Context

The LGTM stack (ADR-0007, ADR-0008) gave us Loki as a log store, but nothing ships Kubernetes container logs into it. The only writer to Loki is the OTel Collector's logs pipeline, whose sole receiver is the external OTLP ingress (otel.<cluster>.<tld>) fed by Claude Code telemetry pushed from developer laptops. There is no in-cluster log agent — no Promtail, no Alloy, no filelog receiver — so workload stdout/stderr (e.g. world-cup-tipping) never reaches Loki and kubectl logs-style querying in Grafana is impossible.

We need an in-cluster agent that tails pod logs from each node and ships them to the existing loki-gateway. Loki already persists to S3 (constellation-loki-<env>, 90-day retention), so the agent itself only needs to read node-local log files and push over HTTP — it needs no AWS access.

Decision

We deploy Grafana Alloy (chart alloy, DaemonSet mode) into the monitoring namespace via ArgoCD, following the same GitOps pattern as the rest of the observability stack. Alloy tails /var/log/pods on each node, parses the CRI log envelope, and pushes to http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push. Loki runs with auth_enabled: false, so no token or IRSA is required — mirroring the OTel Collector, which likewise ships in-cluster over HTTP with zero AWS credentials.

Collection is opt-in: Alloy's discovery.relabel keeps only pods annotated logs.alloy.io/collect: "true". A workload's logs appear in Loki only after its pod template carries that annotation. (Switching to collect-everything later is a one-line change — deleting the keep relabel rule.)

Alloy is deployed as an addition, not a replacement. The OTel Collector (external OTLP gateway), the Prometheus operator, and blackbox-exporter all remain unchanged. Because a log collector must observe every node, Alloy tolerates all taints (operator: Exists) rather than pinning to workload-tier: baseline like other platform components.

Alternatives Considered

Promtail

  • Pros: Purpose-built Loki log shipper; simple; long the default pairing with Loki
  • Cons: Grafana has deprecated Promtail (LTS through Feb 2026, EOL Mar 2026) in favour of Alloy
  • Why not: Adopting a deprecated agent for a brand-new deployment guarantees a near-term migration. Alloy is the supported successor and a superset of Promtail's capabilities.

OTel Collector filelog receiver (DaemonSet)

  • Pros: Keeps the stack on a single telemetry agent; we already run an OTel Collector gateway
  • Cons: The existing OTel Collector is a 2-replica Deployment gateway for external OTLP, a deliberately different role from node-local file tailing; a second OTel Collector in DaemonSet mode with a filelog receiver is less ergonomic and less documented for Kubernetes log collection than Alloy's discovery.kubernetes + loki.source.file components
  • Why not: Alloy is the Grafana-native, best-documented path for pod logs → Loki, and keeps the external-gateway concern cleanly separate from in-cluster collection.

Collect all pod logs automatically (opt-out) instead of opt-in

  • Pros: Zero onboarding friction; logs "just appear" for every workload
  • Cons: Higher Loki/S3 volume and cost from day one; captures noisy/low-value namespaces by default
  • Why not: We chose opt-in for cost control and explicit intent while the platform's log volume is still being characterised. The relabel design makes flipping to opt-out trivial if desired.

FluentBit / Fluentd / Vector

  • Pros: Mature, high-performance log processors
  • Cons: Outside the Grafana ecosystem; another config language and operational model to own; no advantage over Alloy for a Loki-only destination in a Grafana-centric stack
  • Why not: Alloy shares tooling, docs, and mental model with the LGTM stack we already operate.

Consequences

Positive

  • Container logs become queryable in Grafana's Loki datasource with namespace/pod/container/ app labels, enabling trace→log correlation from Tempo (ADR-0008)
  • No Terraform/IRSA/S3 change: Alloy reads node-local files and pushes in-cluster over HTTP
  • Persistence is inherited from Loki→S3; Alloy stays stateless, keeping only a positions/WAL file on a node hostPath for at-least-once delivery across pod restarts
  • Opt-in keeps log volume (and S3 cost) proportional to intent; onboarding is a one-annotation change

Negative

  • One more platform component to operate, and a DaemonSet pod on every node (including app/preview tiers) rather than only baseline nodes
  • Opt-in means teams must remember the annotation; a workload with no annotation silently produces no logs in Loki (mitigated by documenting the annotation in app-onboarding conventions)

Risks

  • The /var/log/pods path layout and CRI log format are runtime-specific; if the node container runtime changes, the __path__ relabel and stage.cri parsing may need revisiting
  • Loki's auth_enabled: false is what allows tokenless push; if multi-tenancy is ever enabled on Loki, Alloy's loki.write will need a tenant header (X-Scope-OrgID), matching the Mimir pattern in ADR-0008