Infra Observability (Kubernetes / declarative)

Grafana 13.2 / Prometheus 3.14 / Loki 3.7 / Tempo 3.0 plus Alloy 1.19, deployed fully declaratively (Kustomize) into the observability namespace of the local orbstack cluster.

Nothing is imperative: no helm install, no dashboard hand-created in the UI. The entire desired state lives in this repository and make deploy reconciles it.

Repository layout

Makefile                              # Every command (make = help)
k8s/
├── kustomization.yaml                # Root Kustomize manifest
├── namespace.yaml                    # observability namespace
├── secret.yaml                       # Grafana admin credentials (GITIGNORED)
├── secret.example.yaml               # Secret template
├── prometheus/
│   ├── prometheus.yaml               # Scrape config (Kubernetes SD)
│   ├── rbac.yaml                     # SA + discovery ClusterRole
│   ├── secret.yaml                   # LibreChat scrape token (GITIGNORED)
│   ├── secret.example.yaml           # Token template
│   ├── pvc.yaml                      # Persistent TSDB (10Gi)
│   ├── deployment.yaml               # Prometheus v3.14.0
│   └── service.yaml                  # :9090
├── loki/
│   ├── loki.yaml                     # Monolithic config, filesystem storage
│   ├── pvc.yaml                      # Chunks + index (10Gi)
│   ├── deployment.yaml               # Loki 3.7.6 (-target=all)
│   └── service.yaml                  # :3100 (HTTP) / :9096 (gRPC)
├── tempo/
│   ├── tempo.yaml                    # Monolithic config + metrics-generator
│   ├── pvc.yaml                      # WAL + trace blocks (10Gi)
│   ├── deployment.yaml               # Tempo 3.0.3 (-target=all)
│   └── service.yaml                  # :3200, OTLP :4317/:4318, Zipkin :9411
├── alloy/
│   ├── config.alloy                  # Pod logs -> Loki pipeline
│   ├── rbac.yaml                     # SA + ClusterRole (including pods/log)
│   ├── daemonset.yaml                # Alloy v1.19.0 (one pod per node)
│   └── service.yaml                  # :12345 (UI + /metrics)
├── grafana/
│   ├── provisioning/
│   │   ├── datasources.yaml          # Datasources with stable UIDs
│   │   └── dashboard-providers.yaml  # File provider
│   ├── dashboards/
│   │   ├── kustomization.yaml        # ConfigMap WITHOUT hash (hot reload)
│   │   ├── obs-stack-health.json     # uid: obs-stack-health
│   │   ├── obs-keycloak.json         # uid: obs-keycloak
│   │   ├── obs-librechat.json        # uid: obs-librechat
│   │   └── obs-classic-models-api.json
│   ├── pvc.yaml                      # Grafana SQLite database (5Gi)
│   ├── deployment.yaml               # Grafana 13.2.0
│   └── service.yaml                  # :3000
└── ingress/
    ├── certificate.yaml              # cert-manager Certificate (sttlab-local-ca)
    └── ingress.yaml                  # Three NGINX Ingresses

Getting started

make                     # Help: every available target
make deploy              # Deploy the whole stack
make hosts               # /etc/hosts entries to add
make grafana-password    # Admin credentials
make smoke               # End-to-end verification

URL	Purpose
https://grafana.sttlab.local	Grafana UI
https://prometheus.sttlab.local	Prometheus UI (targets, expression browser)
https://alloy.sttlab.local	Alloy UI (log pipeline state)

Sending traces from an in-cluster application: tempo.observability.svc.cluster.local:4317 (OTLP gRPC) or :4318 (OTLP HTTP).

Declarative dashboards with stable UIDs

This is the centrepiece of the repository: a dashboard only ever exists as a versioned JSON file, never as an object created through the UI.

Rules

1. One .json file per dashboard, under k8s/grafana/dashboards/.
2. The uid field is stable and never changes. It is what keeps URLs (/d/<uid>/…), cross-dashboard links, annotations and alerts alive across every redeployment. By convention the filename equals the UID (obs-keycloak.json -> uid: obs-keycloak).
3. The id field must stay null (or absent): it is an identifier internal to a Grafana instance and is not portable.
4. Datasources are referenced by stable UID, never by name: "datasource": { "type": "prometheus", "uid": "prometheus" }. Available UIDs: prometheus, loki, tempo.
5. allowUiUpdates: false: any change made in the UI is overwritten on the next reload. The file is the source of truth.

Adding a dashboard

cp my-dashboard.json k8s/grafana/dashboards/obs-my-topic.json
# edit the JSON: "uid": "obs-my-topic", "id": null
make dashboards-sync     # update the list in dashboards/kustomization.yaml
make dashboards          # apply (without restarting Grafana)

make dashboards-check (run automatically by dashboards and validate) verifies: valid JSON, uid present, uid unique, id null, file referenced in the kustomization, filename/UID consistency.

How the hot reload works

The grafana-dashboards ConfigMap is generated without a name-suffix hash (disableNameSuffixHash: true), so make dashboards updates it in place: the Grafana pod is not recreated and the file provider re-reads the directory (updateIntervalSeconds: 10).

Measured end-to-end delay: about 70 seconds (kubelet ConfigMap volume sync ~60s + provider poll 10s). For immediate effect: make restart-grafana.

Conversely, the provisioning ConfigMaps (datasources, providers) keep their hash suffix: those files are only read at startup, so the hash deliberately forces a pod restart when they change.

Persistence

PVC	Component	Size	Contents	Retention
prometheus-data	Prometheus	10Gi	TSDB	15 d / 8 GB
loki-data	Loki	10Gi	chunks + TSDB index	168 h (compactor)
tempo-data	Tempo	10Gi	WAL + trace blocks	168 h
grafana-data	Grafana	5Gi	SQLite database (users, prefs)	—

Applies are server-side. Client-side kubectl apply stores a full copy of each object in the kubectl.kubernetes.io/last-applied-configuration annotation, capped at 256KB; the dashboard ConfigMap passed that in aggregate at eight dashboards. --server-side tracks field ownership in metadata instead, which moves the ceiling to etcd's ~1MB per object. Splitting into one ConfigMap per dashboard is the next step if that is ever reached.

StorageClass local-path (the cluster default), ReadWriteOnce. The Deployments use strategy: Recreate: with an RWO volume the old pod must release it before the new one can start.

Alloy's read positions live on a hostPath (/var/lib/alloy), so restarting the DaemonSet does not re-ingest every log.

Wiping data

make pvc                             # PVC state
make flush-loki       CONFIRM=yes    # wipe the logs only
make flush-prometheus CONFIRM=yes    # wipe the TSDB only
make flush-pvc        CONFIRM=yes    # wipe EVERYTHING

The cycle is: scale 0 -> wait for pod deletion -> delete pvc -> reapply the kustomization (which recreates the PVC) -> scale 1. Every destructive target requires CONFIRM=yes.

Dashboards are not affected by flush-grafana: they are reprovisioned from the ConfigMaps at startup.

Wiring the three signals

• Prometheus discovers its targets through the Kubernetes API: apiserver, kubelet, container resource metrics, plus any Service or Pod carrying the prometheus.io/scrape: "true" annotation (prometheus.io/port, prometheus.io/path).
• Alloy reads every pod's logs through the Kubernetes API (no /var/log mount) and pushes them to Loki with namespace, pod, container, node, app, component and job labels.
• Tempo receives OTLP, and its metrics_generator pushes span metrics and service graphs to Prometheus through its remote-write receiver (--web.enable-remote-write-receiver).
• Grafana correlations: Loki -> Tempo (TraceID derived field), Tempo -> Loki (tracesToLogsV2), Tempo -> Prometheus (serviceMap, tracesToMetrics), Prometheus exemplars -> Tempo.

Container resource metrics

Prometheus scrapes the kubelet's /metrics/resource endpoint, not /metrics/cadvisor. On this cluster the cadvisor endpoint answers with machine_* series only and exposes no container_* at all, so CPU and memory panels would stay empty. /metrics/resource — the source metrics-server itself reads — provides container_cpu_usage_seconds_total and container_memory_working_set_bytes with namespace/pod/container labels.

kube-state-metrics is not deployed, so container restarts are derived from changes in container_start_time_seconds.

Connected applications

Application	Traces	Metrics	Logs	Dashboard
keycloak	OTel Java agent → tempo:4318	native /metrics (management port 9000)	JSON + MDC trace_id	obs-keycloak
librechat	OTel SDK → tempo:4318	built-in /metrics, Bearer token	stdout	obs-librechat
classic-models-api	opentelemetry-instrument → tempo:4318	django-prometheus /metrics	JSON + otelTraceID	obs-classic-models-api

The six Kong gateways share a single dashboard, obs-kong-gateways, with a gateway selector at the top (one, several, or all). Its variable resolves a subtlety of this setup: the metrics label is the full gateway name (catalog-dev) while the audit-log label is the short domain (catalog). The variable holds the short form, PromQL re-adds the -dev suffix, and LogQL uses it as-is — so one control drives both signals. The capture group in its regex also drops the unrelated kong gateway, which carries no -dev suffix.

Their manifests live in the infra-keycloak, infra-librechat and classic-models-api repositories; this repository only holds the platform.

Each application dashboard carries the same four sections: an overview of RED indicators, HTTP traffic, a runtime section specific to the stack in use, and a Logs row, followed by container resources.

Distributed tracing

Traces are searched and explored in Explore, on the Tempo datasource, not from a dashboard. Explore's TraceQL query builder, autocompletion, trace view and interactive service graph have no dashboard equivalent worth maintaining — a dashboard search panel is a worse copy of a tool that already exists.

What the repository does own is the datasource wiring, which applies in Explore exactly as it would in a dashboard.

Span-to-logs correlation

The Tempo datasource resolves a span's logs with a custom query on ${__trace.traceId} rather than by mapping service.name to a Loki label. The label mapping it replaces resolved to app, which Alloy derives from app.kubernetes.io/name: that exists for Keycloak and the Classic Models API but not for the six Kong gateways — and the gateways were the only place the trace id actually appeared. The mapping pointed at the services where it could not work.

Coverage today, each verified by taking a trace id and searching for it:

Service	Trace id in logs	How
Kong gateways (×6)	yes	x-trace-id in the http-log record
Keycloak	yes	MDC, via KC_LOG_MDC_ENABLED
classic-models-api	yes	inbound traceparent in the gunicorn access line
LibreChat	no	would need the upstream image patched

Span-to-metrics correlation

tracesToMetrics had no queries defined and rendered a bare datasource link. It now carries rate, error-rate and p95 queries against traces_spanmetrics_*, scoped to the span's own service.

Those span metrics are worth knowing about independently: Tempo's metrics_generator derives them from the traces and remote-writes them to Prometheus, keyed by service and span_name. They measure the span, so a slow database statement appears next to a slow HTTP handler — a granularity the gateway and per-application dashboards, which measure HTTP, do not have.

Log dashboards, split by kind

Logs are searched from three dashboards rather than one, because the three questions have nothing in common and a single view compromised on all of them.

Dashboard	Question	Sources
obs-access-logs	Which HTTP requests came in?	Kong records, Keycloak access logger, Django gunicorn, ingress-nginx
obs-application-logs	What is the code reporting, and what broke?	Keycloak, LibreChat, Django, the Kong dataplanes' nginx error log
obs-audit-logs	Who authenticated, what changed, who called what?	Keycloak org.keycloak.events, Kong http-log records

Each source keeps its own parser and they all emit the same field names, so the frames merge into one table with consistent columns.

Keycloak's identity events are read with one independent regexp per field rather than line_format piped into logfmt. Two reasons, both measured: that combination is pathological on this data — each stage answers in 0.3s alone, chained they time out at 60s and the table never renders — and the field set differs per event type, so a single regexp spanning them would fail whole rows on a missing field. Independent stages leave the absent field empty and keep the row, and the query returns in 0.1s. Access logs converge on method / path / status / client; application logs on level / message.

Three things this split fixes that a single consolidated view could not:

The kinds are subtracted, not just filtered. Application logs exclude Keycloak's access-log and events loggers by name, the Django container's NCSA lines by shape, and the Kong containers' stray access lines by requiring the nginx date prefix. Without that, access lines drown the signal.

Levels are read from the line, not inferred. Loki's detected_level marks 78% of the Kong gateway output unknown even though every one of those lines states [notice] or [warn]. Each parser extracts the level and lowercases it, so nginx's notice, Keycloak's INFO and LibreChat's info land on one scale — and the level selector has no unknown entry, because a line reaching the table always states one.

The scope selector actually scopes. Each query carries both job=~"${source:regex}" and a literal job matcher. With a single selector, choosing "All" widened the query back to every stream in the cluster — which is how MongoDB and cert-manager ended up in an application view.

Latency is populated for Kong only: the NCSA format the other sources use carries no timing field. That is a property of the format, not a gap in the query.

The NCSA sources are read with regexp, not pattern. A trailing pattern capture consumes the rest of the line, so a <size> at the end swallowed the referer and user-agent along with the byte count. Bounding the capture on \d+ stops it at the number, and regexp tolerates both shapes — Keycloak's access line ends at the size, gunicorn's and nginx's carry more fields after it.

In every table the raw record is the last column, and the organize transformation's indexByName keys on the original field names rather than the renamed labels — keying it on the display names silently leaves the column order to whatever order the frames arrived in.

Parsed log tables### Parsed log tables

Every dashboard shows its logs twice: a table with one column per parsed field, and, in a collapsed row underneath, the original logs panel.

The table is what makes a JSON record readable — status, method, route, latency as sortable, filterable columns rather than a wall of braces. Its last column holds the unparsed record, and every cell has the inspector enabled, so the raw line is one click away.

The logs panel stays because a table cannot do what it does: render ANSI colour, expand a record inline, tail live, and turn a trace_id into a link to Tempo. It is collapsed so it costs nothing until needed.

Parsing is per-format, because these five sources agree on nothing:

Source	Format	Parser
Kong audit	nested JSON	| json with explicit field paths
Keycloak	JSON	| json
classic-models-api	NCSA access lines	| pattern
LibreChat	ANSI-coloured text	line_format to strip ANSI, then | pattern
Observability stack	logfmt	| logfmt with named fields

Two choices worth knowing. The Kong and stack parsers name their fields explicitly rather than extracting everything: a bare \| json on a Kong record yields about seventy columns, and a bare \| logfmt explodes Loki's own query-metrics lines. And \| pattern keeps lines it cannot match, with empty columns, rather than dropping them — so the Django table never silently hides an application log among the access lines.

The Logs row leans on Loki's detected_level, which is inferred at ingestion time, so a single query works across all three log formats — Keycloak emits JSON, LibreChat colour-coded text, and the Django API plain gunicorn access lines. No per-application parser is needed. The volume panel is scoped to the application container, while the two error panels cover the whole namespace so the datastore's own failures surface alongside the application's. Where a log line carries a trace_id, the Loki datasource's derived field turns it into a link straight to the matching trace in Tempo.

Connecting an application

1. Traces — point OTLP at Tempo: OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.observability.svc.cluster.local:4318 (:4317 for gRPC). RED span metrics and the service graph are then derived automatically by Tempo's metrics_generator, with no extra instrumentation.
2. Metrics — annotate the pod (not the Service):

   annotations:
     prometheus.io/scrape: "true"
     prometheus.io/port: "9000"
     prometheus.io/path: "/metrics"
   

3. Logs — nothing to do: Alloy collects every pod in the cluster. For Grafana to jump from a log line to its trace, the line only needs to contain a trace_id (the Loki datasource's derived field).

Token-protected endpoints

Annotation-based discovery cannot carry credentials, so an authenticated target gets its own scrape_config in k8s/prometheus/prometheus.yaml, with the token mounted from a Secret — see the librechat job, which reads /etc/prometheus/secrets/librechat/token.

That token must be identical on both sides: METRICS_SECRET in the librechat-secrets secret (namespace librechat) and the token key of the librechat-metrics secret (namespace observability, file k8s/prometheus/secret.yaml, gitignored).

A Kubernetes Secret is namespaced and a pod can only mount ones from its own namespace, so the same value has to exist twice — once for LibreChat, which verifies the token, once for Prometheus, which presents it. The risk is silent drift: regenerating one side without the other sends the scrape back to 401 without breaking anything else. make check-metrics-token (included in make diag) compares both copies.

Diagnostics

make status              # pods, services, PVCs, ingresses
make diag                # + endpoints, certificate, failing containers, events
make smoke               # health of all five components, datasources, dashboards, targets
make targets             # Prometheus targets and their last scrape error
make logs-loki           # follow one component's logs
make top                 # CPU/memory usage
make pf-grafana          # port-forward when the ingress is unavailable
make diff                # difference between the repository and the cluster
make validate            # kustomize build + server dry-run + dashboard checks

Secrets

k8s/secret.yaml and k8s/prometheus/secret.yaml are gitignored. make secret generates the first from secret.example.yaml with a random admin password if it does not exist (and is called automatically by make deploy).

The TLS certificate is not versioned either: it is issued by the existing sttlab-local-ca cert-manager ClusterIssuer (k8s/ingress/certificate.yaml).

Coexisting with what was already there

The observability namespace already hosts two deployments not managed by this repository, left untouched:

• jaeger (installed by Helm) — it also exposes OTLP on 4317/4318, but on its own Service, so there is no conflict with Tempo. Since all three applications now point at Tempo, Jaeger no longer receives anything; uninstall the Helm release whenever you want a single trace backend.
• audit-log-collector (Fluent Bit) — writes to its own PVC.

make undeploy and make destroy only run kubectl delete -k k8s/, so they touch nothing but the resources described here.