Stéphane Tailland
d3abde1040
The dashboard reimplemented, less well, what Explore already does: its search panel had no TraceQL query builder and no autocompletion, and its service dependency table replaced an interactive node graph with a list. Searching and exploring traces belongs in Explore. The datasource correlations this work also fixed are kept — they are what was actually missing, and they apply in Explore exactly as they would in a dashboard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
426 lines
21 KiB
Markdown
426 lines
21 KiB
Markdown
# Infra Observability (Kubernetes / declarative)
|
||
|
||
**Grafana 13.2 / Prometheus 3.14 / Loki 3.7 / Tempo 3.0** plus **Alloy 1.19**,
|
||
deployed fully declaratively (Kustomize) into the `observability` namespace of
|
||
the local `orbstack` cluster.
|
||
|
||
Nothing is imperative: no `helm install`, no dashboard hand-created in the UI.
|
||
The entire desired state lives in this repository and `make deploy` reconciles it.
|
||
|
||
## Repository layout
|
||
|
||
```text
|
||
Makefile # Every command (make = help)
|
||
k8s/
|
||
├── kustomization.yaml # Root Kustomize manifest
|
||
├── namespace.yaml # observability namespace
|
||
├── secret.yaml # Grafana admin credentials (GITIGNORED)
|
||
├── secret.example.yaml # Secret template
|
||
├── prometheus/
|
||
│ ├── prometheus.yaml # Scrape config (Kubernetes SD)
|
||
│ ├── rbac.yaml # SA + discovery ClusterRole
|
||
│ ├── secret.yaml # LibreChat scrape token (GITIGNORED)
|
||
│ ├── secret.example.yaml # Token template
|
||
│ ├── pvc.yaml # Persistent TSDB (10Gi)
|
||
│ ├── deployment.yaml # Prometheus v3.14.0
|
||
│ └── service.yaml # :9090
|
||
├── loki/
|
||
│ ├── loki.yaml # Monolithic config, filesystem storage
|
||
│ ├── pvc.yaml # Chunks + index (10Gi)
|
||
│ ├── deployment.yaml # Loki 3.7.6 (-target=all)
|
||
│ └── service.yaml # :3100 (HTTP) / :9096 (gRPC)
|
||
├── tempo/
|
||
│ ├── tempo.yaml # Monolithic config + metrics-generator
|
||
│ ├── pvc.yaml # WAL + trace blocks (10Gi)
|
||
│ ├── deployment.yaml # Tempo 3.0.3 (-target=all)
|
||
│ └── service.yaml # :3200, OTLP :4317/:4318, Zipkin :9411
|
||
├── alloy/
|
||
│ ├── config.alloy # Pod logs -> Loki pipeline
|
||
│ ├── rbac.yaml # SA + ClusterRole (including pods/log)
|
||
│ ├── daemonset.yaml # Alloy v1.19.0 (one pod per node)
|
||
│ └── service.yaml # :12345 (UI + /metrics)
|
||
├── grafana/
|
||
│ ├── provisioning/
|
||
│ │ ├── datasources.yaml # Datasources with stable UIDs
|
||
│ │ └── dashboard-providers.yaml # File provider
|
||
│ ├── dashboards/
|
||
│ │ ├── kustomization.yaml # ConfigMap WITHOUT hash (hot reload)
|
||
│ │ ├── obs-stack-health.json # uid: obs-stack-health
|
||
│ │ ├── obs-keycloak.json # uid: obs-keycloak
|
||
│ │ ├── obs-librechat.json # uid: obs-librechat
|
||
│ │ └── obs-classic-models-api.json
|
||
│ ├── pvc.yaml # Grafana SQLite database (5Gi)
|
||
│ ├── deployment.yaml # Grafana 13.2.0
|
||
│ └── service.yaml # :3000
|
||
└── ingress/
|
||
├── certificate.yaml # cert-manager Certificate (sttlab-local-ca)
|
||
└── ingress.yaml # Three NGINX Ingresses
|
||
```
|
||
|
||
## Getting started
|
||
|
||
```bash
|
||
make # Help: every available target
|
||
make deploy # Deploy the whole stack
|
||
make hosts # /etc/hosts entries to add
|
||
make grafana-password # Admin credentials
|
||
make smoke # End-to-end verification
|
||
```
|
||
|
||
| URL | Purpose |
|
||
|---|---|
|
||
| `https://grafana.sttlab.local` | Grafana UI |
|
||
| `https://prometheus.sttlab.local` | Prometheus UI (targets, expression browser) |
|
||
| `https://alloy.sttlab.local` | Alloy UI (log pipeline state) |
|
||
|
||
Sending traces from an in-cluster application:
|
||
`tempo.observability.svc.cluster.local:4317` (OTLP gRPC) or `:4318` (OTLP HTTP).
|
||
|
||
## Declarative dashboards with stable UIDs
|
||
|
||
This is the centrepiece of the repository: **a dashboard only ever exists as a
|
||
versioned JSON file**, never as an object created through the UI.
|
||
|
||
### Rules
|
||
|
||
1. **One `.json` file per dashboard**, under `k8s/grafana/dashboards/`.
|
||
2. **The `uid` field is stable and never changes.** It is what keeps URLs
|
||
(`/d/<uid>/…`), cross-dashboard links, annotations and alerts alive across
|
||
every redeployment. By convention the filename equals the UID
|
||
(`obs-keycloak.json` -> `uid: obs-keycloak`).
|
||
3. **The `id` field must stay `null`** (or absent): it is an identifier internal
|
||
to a Grafana instance and is not portable.
|
||
4. **Datasources are referenced by stable UID**, never by name:
|
||
`"datasource": { "type": "prometheus", "uid": "prometheus" }`.
|
||
Available UIDs: `prometheus`, `loki`, `tempo`.
|
||
5. `allowUiUpdates: false`: any change made in the UI is overwritten on the next
|
||
reload. The file is the source of truth.
|
||
|
||
### Adding a dashboard
|
||
|
||
```bash
|
||
cp my-dashboard.json k8s/grafana/dashboards/obs-my-topic.json
|
||
# edit the JSON: "uid": "obs-my-topic", "id": null
|
||
make dashboards-sync # update the list in dashboards/kustomization.yaml
|
||
make dashboards # apply (without restarting Grafana)
|
||
```
|
||
|
||
`make dashboards-check` (run automatically by `dashboards` and `validate`)
|
||
verifies: valid JSON, `uid` present, `uid` unique, `id` null, file referenced in
|
||
the kustomization, filename/UID consistency.
|
||
|
||
### How the hot reload works
|
||
|
||
The `grafana-dashboards` ConfigMap is generated **without a name-suffix hash**
|
||
(`disableNameSuffixHash: true`), so `make dashboards` updates it *in place*: the
|
||
Grafana pod is not recreated and the file provider re-reads the directory
|
||
(`updateIntervalSeconds: 10`).
|
||
|
||
Measured end-to-end delay: **about 70 seconds** (kubelet ConfigMap volume sync
|
||
~60s + provider poll 10s). For immediate effect: `make restart-grafana`.
|
||
|
||
Conversely, the *provisioning* ConfigMaps (datasources, providers) **keep** their
|
||
hash suffix: those files are only read at startup, so the hash deliberately
|
||
forces a pod restart when they change.
|
||
|
||
## Persistence
|
||
|
||
| PVC | Component | Size | Contents | Retention |
|
||
|---|---|---|---|---|
|
||
| `prometheus-data` | Prometheus | 10Gi | TSDB | 15 d / 8 GB |
|
||
| `loki-data` | Loki | 10Gi | chunks + TSDB index | 168 h (compactor) |
|
||
| `tempo-data` | Tempo | 10Gi | WAL + trace blocks | 168 h |
|
||
| `grafana-data` | Grafana | 5Gi | SQLite database (users, prefs) | — |
|
||
|
||
> **Applies are server-side.** Client-side `kubectl apply` stores a full copy of
|
||
> each object in the `kubectl.kubernetes.io/last-applied-configuration`
|
||
> annotation, capped at 256KB; the dashboard ConfigMap passed that in aggregate
|
||
> at eight dashboards. `--server-side` tracks field ownership in metadata
|
||
> instead, which moves the ceiling to etcd's ~1MB per object. Splitting into one
|
||
> ConfigMap per dashboard is the next step if that is ever reached.
|
||
|
||
StorageClass `local-path` (the cluster default), `ReadWriteOnce`. The Deployments
|
||
use `strategy: Recreate`: with an RWO volume the old pod must release it before
|
||
the new one can start.
|
||
|
||
Alloy's read positions live on a `hostPath` (`/var/lib/alloy`), so restarting the
|
||
DaemonSet does not re-ingest every log.
|
||
|
||
### Wiping data
|
||
|
||
```bash
|
||
make pvc # PVC state
|
||
make flush-loki CONFIRM=yes # wipe the logs only
|
||
make flush-prometheus CONFIRM=yes # wipe the TSDB only
|
||
make flush-pvc CONFIRM=yes # wipe EVERYTHING
|
||
```
|
||
|
||
The cycle is: `scale 0` -> wait for pod deletion -> `delete pvc` -> reapply the
|
||
kustomization (which recreates the PVC) -> `scale 1`. Every destructive target
|
||
requires `CONFIRM=yes`.
|
||
|
||
Dashboards are **not** affected by `flush-grafana`: they are reprovisioned from
|
||
the ConfigMaps at startup.
|
||
|
||
## Wiring the three signals
|
||
|
||
- **Prometheus** discovers its targets through the Kubernetes API: apiserver,
|
||
kubelet, container resource metrics, plus any Service or Pod carrying the
|
||
`prometheus.io/scrape: "true"` annotation (`prometheus.io/port`,
|
||
`prometheus.io/path`).
|
||
- **Alloy** reads every pod's logs through the Kubernetes API (no `/var/log`
|
||
mount) and pushes them to Loki with `namespace`, `pod`, `container`, `node`,
|
||
`app`, `component` and `job` labels.
|
||
- **Tempo** receives OTLP, and its `metrics_generator` pushes span metrics and
|
||
service graphs to Prometheus through its remote-write receiver
|
||
(`--web.enable-remote-write-receiver`).
|
||
- **Grafana correlations**: Loki -> Tempo (`TraceID` derived field),
|
||
Tempo -> Loki (`tracesToLogsV2`), Tempo -> Prometheus (`serviceMap`,
|
||
`tracesToMetrics`), Prometheus exemplars -> Tempo.
|
||
|
||
### Container resource metrics
|
||
|
||
Prometheus scrapes the kubelet's **`/metrics/resource`** endpoint, not
|
||
`/metrics/cadvisor`. On this cluster the cadvisor endpoint answers with `machine_*`
|
||
series only and exposes no `container_*` at all, so CPU and memory panels would
|
||
stay empty. `/metrics/resource` — the source metrics-server itself reads —
|
||
provides `container_cpu_usage_seconds_total` and
|
||
`container_memory_working_set_bytes` with `namespace`/`pod`/`container` labels.
|
||
|
||
kube-state-metrics is not deployed, so container restarts are derived from
|
||
changes in `container_start_time_seconds`.
|
||
|
||
## Connected applications
|
||
|
||
| Application | Traces | Metrics | Logs | Dashboard |
|
||
|---|---|---|---|---|
|
||
| `keycloak` | OTel Java agent → `tempo:4318` | native `/metrics` (management port 9000) | JSON + MDC `trace_id` | `obs-keycloak` |
|
||
| `librechat` | OTel SDK → `tempo:4318` | built-in `/metrics`, Bearer token | stdout | `obs-librechat` |
|
||
| `classic-models-api` | `opentelemetry-instrument` → `tempo:4318` | django-prometheus `/metrics` | JSON + `otelTraceID` | `obs-classic-models-api` |
|
||
|
||
The six Kong gateways share a single dashboard, `obs-kong-gateways`, with a
|
||
`gateway` selector at the top (one, several, or all). Its variable resolves a
|
||
subtlety of this setup: the metrics label is the full gateway name
|
||
(`catalog-dev`) while the audit-log label is the short domain (`catalog`). The
|
||
variable holds the short form, PromQL re-adds the `-dev` suffix, and LogQL uses
|
||
it as-is — so one control drives both signals. The capture group in its regex
|
||
also drops the unrelated `kong` gateway, which carries no `-dev` suffix.
|
||
|
||
Their manifests live in the `infra-keycloak`, `infra-librechat` and
|
||
`classic-models-api` repositories; this repository only holds the platform.
|
||
|
||
Each application dashboard carries the same four sections: an overview of RED
|
||
indicators, HTTP traffic, a runtime section specific to the stack in use, and a
|
||
**Logs** row, followed by container resources.
|
||
|
||
### Distributed tracing
|
||
|
||
Traces are searched and explored in **Explore**, on the Tempo datasource, not
|
||
from a dashboard. Explore's TraceQL query builder, autocompletion, trace view
|
||
and interactive service graph have no dashboard equivalent worth maintaining —
|
||
a dashboard search panel is a worse copy of a tool that already exists.
|
||
|
||
What the repository does own is the datasource wiring, which applies in Explore
|
||
exactly as it would in a dashboard.
|
||
|
||
#### Span-to-logs correlation
|
||
|
||
The Tempo datasource resolves a span's logs with a custom query on
|
||
`${__trace.traceId}` rather than by mapping `service.name` to a Loki label. The
|
||
label mapping it replaces resolved to `app`, which Alloy derives from
|
||
`app.kubernetes.io/name`: that exists for Keycloak and the Classic Models API
|
||
but not for the six Kong gateways — and the gateways were the only place the
|
||
trace id actually appeared. The mapping pointed at the services where it could
|
||
not work.
|
||
|
||
Coverage today, each verified by taking a trace id and searching for it:
|
||
|
||
| Service | Trace id in logs | How |
|
||
|---|---|---|
|
||
| Kong gateways (×6) | yes | `x-trace-id` in the http-log record |
|
||
| Keycloak | yes | MDC, via `KC_LOG_MDC_ENABLED` |
|
||
| classic-models-api | yes | inbound `traceparent` in the gunicorn access line |
|
||
| LibreChat | no | would need the upstream image patched |
|
||
|
||
#### Span-to-metrics correlation
|
||
|
||
`tracesToMetrics` had no queries defined and rendered a bare datasource link. It
|
||
now carries rate, error-rate and p95 queries against `traces_spanmetrics_*`,
|
||
scoped to the span's own service.
|
||
|
||
Those span metrics are worth knowing about independently: Tempo's
|
||
`metrics_generator` derives them from the traces and remote-writes them to
|
||
Prometheus, keyed by `service` and `span_name`. They measure the *span*, so a
|
||
slow database statement appears next to a slow HTTP handler — a granularity the
|
||
gateway and per-application dashboards, which measure HTTP, do not have.
|
||
|
||
### Log dashboards, split by kind
|
||
|
||
Logs are searched from three dashboards rather than one, because the three
|
||
questions have nothing in common and a single view compromised on all of them.
|
||
|
||
| Dashboard | Question | Sources |
|
||
|---|---|---|
|
||
| `obs-access-logs` | Which HTTP requests came in? | Kong records, Keycloak access logger, Django gunicorn, ingress-nginx |
|
||
| `obs-application-logs` | What is the code reporting, and what broke? | Keycloak, LibreChat, Django, the Kong dataplanes' nginx error log |
|
||
| `obs-audit-logs` | Who authenticated, what changed, who called what? | Keycloak `org.keycloak.events`, Kong http-log records |
|
||
|
||
Each source keeps its own parser and they all emit the same field names, so the
|
||
frames merge into one table with consistent columns.
|
||
|
||
Keycloak's identity events are read with one independent `regexp` per field
|
||
rather than `line_format` piped into `logfmt`. Two reasons, both measured: that
|
||
combination is pathological on this data — each stage answers in 0.3s alone,
|
||
chained they time out at 60s and the table never renders — and the field set
|
||
differs per event type, so a single regexp spanning them would fail whole rows
|
||
on a missing field. Independent stages leave the absent field empty and keep the
|
||
row, and the query returns in 0.1s. Access logs converge on
|
||
method / path / status / client; application logs on level / message.
|
||
|
||
Three things this split fixes that a single consolidated view could not:
|
||
|
||
**The kinds are subtracted, not just filtered.** Application logs exclude
|
||
Keycloak's `access-log` and `events` loggers by name, the Django container's
|
||
NCSA lines by shape, and the Kong containers' stray access lines by requiring
|
||
the nginx date prefix. Without that, access lines drown the signal.
|
||
|
||
**Levels are read from the line, not inferred.** Loki's `detected_level` marks
|
||
78% of the Kong gateway output `unknown` even though every one of those lines
|
||
states `[notice]` or `[warn]`. Each parser extracts the level and lowercases it,
|
||
so nginx's `notice`, Keycloak's `INFO` and LibreChat's `info` land on one scale
|
||
— and the level selector has no `unknown` entry, because a line reaching the
|
||
table always states one.
|
||
|
||
**The scope selector actually scopes.** Each query carries both
|
||
`job=~"${source:regex}"` and a literal job matcher. With a single selector,
|
||
choosing "All" widened the query back to every stream in the cluster — which is
|
||
how MongoDB and cert-manager ended up in an application view.
|
||
|
||
Latency is populated for Kong only: the NCSA format the other sources use
|
||
carries no timing field. That is a property of the format, not a gap in the
|
||
query.
|
||
|
||
The NCSA sources are read with `regexp`, not `pattern`. A trailing `pattern`
|
||
capture consumes the rest of the line, so a `<size>` at the end swallowed the
|
||
referer and user-agent along with the byte count. Bounding the capture on `\d+`
|
||
stops it at the number, and `regexp` tolerates both shapes — Keycloak's access
|
||
line ends at the size, gunicorn's and nginx's carry more fields after it.
|
||
|
||
In every table the raw record is the last column, and the `organize`
|
||
transformation's `indexByName` keys on the original field names rather than the
|
||
renamed labels — keying it on the display names silently leaves the column order
|
||
to whatever order the frames arrived in.
|
||
|
||
### Parsed log tables### Parsed log tables
|
||
|
||
Every dashboard shows its logs twice: a **table** with one column per parsed
|
||
field, and, in a collapsed row underneath, the original **logs panel**.
|
||
|
||
The table is what makes a JSON record readable — status, method, route, latency
|
||
as sortable, filterable columns rather than a wall of braces. Its last column
|
||
holds the unparsed record, and every cell has the inspector enabled, so the raw
|
||
line is one click away.
|
||
|
||
The logs panel stays because a table cannot do what it does: render ANSI colour,
|
||
expand a record inline, tail live, and turn a `trace_id` into a link to Tempo.
|
||
It is collapsed so it costs nothing until needed.
|
||
|
||
Parsing is per-format, because these five sources agree on nothing:
|
||
|
||
| Source | Format | Parser |
|
||
|---|---|---|
|
||
| Kong audit | nested JSON | `\| json` with explicit field paths |
|
||
| Keycloak | JSON | `\| json` |
|
||
| classic-models-api | NCSA access lines | `\| pattern` |
|
||
| LibreChat | ANSI-coloured text | `line_format` to strip ANSI, then `\| pattern` |
|
||
| Observability stack | logfmt | `\| logfmt` with named fields |
|
||
|
||
Two choices worth knowing. The Kong and stack parsers name their fields
|
||
explicitly rather than extracting everything: a bare `\| json` on a Kong record
|
||
yields about seventy columns, and a bare `\| logfmt` explodes Loki's own
|
||
query-metrics lines. And `\| pattern` keeps lines it cannot match, with empty
|
||
columns, rather than dropping them — so the Django table never silently hides
|
||
an application log among the access lines.
|
||
|
||
The Logs row leans on Loki's `detected_level`, which is inferred at ingestion
|
||
time, so a single query works across all three log formats — Keycloak emits JSON,
|
||
LibreChat colour-coded text, and the Django API plain gunicorn access lines. No
|
||
per-application parser is needed. The volume panel is scoped to the application
|
||
container, while the two error panels cover the whole namespace so the
|
||
datastore's own failures surface alongside the application's. Where a log line
|
||
carries a `trace_id`, the Loki datasource's derived field turns it into a link
|
||
straight to the matching trace in Tempo.
|
||
|
||
### Connecting an application
|
||
|
||
1. **Traces** — point OTLP at Tempo:
|
||
`OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.observability.svc.cluster.local:4318`
|
||
(`:4317` for gRPC). RED span metrics and the service graph are then derived
|
||
automatically by Tempo's `metrics_generator`, with no extra instrumentation.
|
||
2. **Metrics** — annotate the **pod** (not the Service):
|
||
```yaml
|
||
annotations:
|
||
prometheus.io/scrape: "true"
|
||
prometheus.io/port: "9000"
|
||
prometheus.io/path: "/metrics"
|
||
```
|
||
3. **Logs** — nothing to do: Alloy collects every pod in the cluster. For Grafana
|
||
to jump from a log line to its trace, the line only needs to contain a
|
||
`trace_id` (the Loki datasource's derived field).
|
||
|
||
### Token-protected endpoints
|
||
|
||
Annotation-based discovery cannot carry credentials, so an authenticated target
|
||
gets its own `scrape_config` in `k8s/prometheus/prometheus.yaml`, with the token
|
||
mounted from a Secret — see the `librechat` job, which reads
|
||
`/etc/prometheus/secrets/librechat/token`.
|
||
|
||
That token must be **identical** on both sides: `METRICS_SECRET` in the
|
||
`librechat-secrets` secret (namespace `librechat`) and the `token` key of the
|
||
`librechat-metrics` secret (namespace `observability`, file
|
||
`k8s/prometheus/secret.yaml`, gitignored).
|
||
|
||
A Kubernetes Secret is namespaced and a pod can only mount ones from its own
|
||
namespace, so the same value has to exist twice — once for LibreChat, which
|
||
**verifies** the token, once for Prometheus, which **presents** it. The risk is
|
||
silent drift: regenerating one side without the other sends the scrape back to
|
||
401 without breaking anything else. `make check-metrics-token` (included in
|
||
`make diag`) compares both copies.
|
||
|
||
## Diagnostics
|
||
|
||
```bash
|
||
make status # pods, services, PVCs, ingresses
|
||
make diag # + endpoints, certificate, failing containers, events
|
||
make smoke # health of all five components, datasources, dashboards, targets
|
||
make targets # Prometheus targets and their last scrape error
|
||
make logs-loki # follow one component's logs
|
||
make top # CPU/memory usage
|
||
make pf-grafana # port-forward when the ingress is unavailable
|
||
make diff # difference between the repository and the cluster
|
||
make validate # kustomize build + server dry-run + dashboard checks
|
||
```
|
||
|
||
## Secrets
|
||
|
||
`k8s/secret.yaml` and `k8s/prometheus/secret.yaml` are **gitignored**.
|
||
`make secret` generates the first from `secret.example.yaml` with a random admin
|
||
password if it does not exist (and is called automatically by `make deploy`).
|
||
|
||
The TLS certificate is not versioned either: it is issued by the existing
|
||
`sttlab-local-ca` cert-manager ClusterIssuer (`k8s/ingress/certificate.yaml`).
|
||
|
||
## Coexisting with what was already there
|
||
|
||
The `observability` namespace already hosts two deployments **not managed by this
|
||
repository**, left untouched:
|
||
|
||
- `jaeger` (installed by Helm) — it also exposes OTLP on `4317`/`4318`, but on its
|
||
own Service, so there is no conflict with Tempo. Since all three applications
|
||
now point at Tempo, Jaeger no longer receives anything; uninstall the Helm
|
||
release whenever you want a single trace backend.
|
||
- `audit-log-collector` (Fluent Bit) — writes to its own PVC.
|
||
|
||
`make undeploy` and `make destroy` only run `kubectl delete -k k8s/`, so they
|
||
touch nothing but the resources described here.
|