Declarative Grafana / Prometheus / Loki / Tempo observability stack

Deploys the whole stack into the observability namespace of the local
cluster with Kustomize alone — no helm install, no dashboard created by
hand in the UI. The Makefile carries every command: deploy, dashboards,
diagnostics, PVC flush, teardown.

Dashboards are files, never UI objects. Each carries a stable uid, which
is what keeps URLs, cross-links and alerts alive across redeployments, and
references datasources by stable uid rather than by name. The dashboard
ConfigMap is generated without a name-suffix hash so applying it updates
it in place and Grafana reloads without a restart, while the provisioning
ConfigMaps keep their hash precisely so that a change there does force one.

Prometheus scrapes the kubelet /metrics/resource endpoint rather than
/metrics/cadvisor: on this cluster cadvisor exposes machine_* series only
and no container_* at all, which would leave every CPU and memory panel
empty.

Alloy ships every pod's logs to Loki through the Kubernetes API, so no
/var/log mount is needed. Tempo's metrics_generator feeds span metrics and
the service graph back into Prometheus, and the datasources are wired so a
log line's trace_id opens the matching trace.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/grafana/dashboards/kustomization.yaml

@@ -0,0 +1,31 @@
+# Grafana dashboards -- declarative provisioning.
+#
+# Rules:
+#   * one .json file per dashboard, named after its UID;
+#   * every JSON carries a STABLE "uid" field (never regenerated): that is what
+#     keeps URLs, cross-links and alerts alive across redeployments;
+#   * `disableNameSuffixHash: true` -> the ConfigMap always keeps the same name,
+#     so `kubectl apply` updates it in place and Grafana reloads WITHOUT
+#     restarting the pod, in about 70s (see dashboard-providers.yaml).
+#
+# Adding a dashboard = drop the .json here and add it to `files:` below.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: observability
+
+configMapGenerator:
+  - name: grafana-dashboards
+    files:
+      # >>> dashboards (generated by `make dashboards-sync`) >>>
+      - obs-classic-models-api.json
+      - obs-keycloak.json
+      - obs-librechat.json
+      - obs-stack-health.json
+      # <<< dashboards <<<
+
+generatorOptions:
+  disableNameSuffixHash: true
+  labels:
+    app.kubernetes.io/name: grafana
+    app.kubernetes.io/component: dashboards
+    app.kubernetes.io/part-of: observability