Declarative Grafana / Prometheus / Loki / Tempo observability stack

Deploys the whole stack into the observability namespace of the local
cluster with Kustomize alone — no helm install, no dashboard created by
hand in the UI. The Makefile carries every command: deploy, dashboards,
diagnostics, PVC flush, teardown.

Dashboards are files, never UI objects. Each carries a stable uid, which
is what keeps URLs, cross-links and alerts alive across redeployments, and
references datasources by stable uid rather than by name. The dashboard
ConfigMap is generated without a name-suffix hash so applying it updates
it in place and Grafana reloads without a restart, while the provisioning
ConfigMaps keep their hash precisely so that a change there does force one.

Prometheus scrapes the kubelet /metrics/resource endpoint rather than
/metrics/cadvisor: on this cluster cadvisor exposes machine_* series only
and no container_* at all, which would leave every CPU and memory panel
empty.

Alloy ships every pod's logs to Loki through the Kubernetes API, so no
/var/log mount is needed. Tempo's metrics_generator feeds span metrics and
the service graph back into Prometheus, and the datasources are wired so a
log line's trace_id opens the matching trace.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/alloy/rbac.yaml

@@ -0,0 +1,52 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: alloy
+  namespace: observability
+  labels:
+    app.kubernetes.io/name: alloy
+    app.kubernetes.io/part-of: observability
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: observability-alloy
+  labels:
+    app.kubernetes.io/name: alloy
+    app.kubernetes.io/part-of: observability
+rules:
+  - apiGroups: [""]
+    resources:
+      - nodes
+      - nodes/proxy
+      - nodes/metrics
+      - services
+      - endpoints
+      - pods
+      - namespaces
+      - events
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["pods/log"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["discovery.k8s.io"]
+    resources: ["endpointslices"]
+    verbs: ["get", "list", "watch"]
+  - nonResourceURLs: ["/metrics"]
+    verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: observability-alloy
+  labels:
+    app.kubernetes.io/name: alloy
+    app.kubernetes.io/part-of: observability
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: observability-alloy
+subjects:
+  - kind: ServiceAccount
+    name: alloy
+    namespace: observability