Declarative Grafana / Prometheus / Loki / Tempo observability stack

Deploys the whole stack into the observability namespace of the local
cluster with Kustomize alone — no helm install, no dashboard created by
hand in the UI. The Makefile carries every command: deploy, dashboards,
diagnostics, PVC flush, teardown.

Dashboards are files, never UI objects. Each carries a stable uid, which
is what keeps URLs, cross-links and alerts alive across redeployments, and
references datasources by stable uid rather than by name. The dashboard
ConfigMap is generated without a name-suffix hash so applying it updates
it in place and Grafana reloads without a restart, while the provisioning
ConfigMaps keep their hash precisely so that a change there does force one.

Prometheus scrapes the kubelet /metrics/resource endpoint rather than
/metrics/cadvisor: on this cluster cadvisor exposes machine_* series only
and no container_* at all, which would leave every CPU and memory panel
empty.

Alloy ships every pod's logs to Loki through the Kubernetes API, so no
/var/log mount is needed. Tempo's metrics_generator feeds span metrics and
the service graph back into Prometheus, and the datasources are wired so a
log line's trace_id opens the matching trace.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This commit is contained in:

Stéphane Tailland

2026-08-25 07:27:38 +02:00

commit 3937b4f366

42 changed files with 7597 additions and 0 deletions

									
										k8s/secret.example.yaml
									
		+11
		
												View File
												
				@@ -0,0 +1,11 @@

				# Template: copy to secret.yaml (gitignored) and fill in the values.

				#   make secret

				apiVersion: v1

				kind: Secret

				metadata:

				  name: observability-secrets

				  namespace: observability

				type: Opaque

				stringData:

				  GF_SECURITY_ADMIN_USER: admin

				  GF_SECURITY_ADMIN_PASSWORD: <CHANGE_ME>