infra-observability/README.md

# Infra Observability (Kubernetes / declarative)

**Grafana 13.2 / Prometheus 3.14 / Loki 3.7 / Tempo 3.0** plus **Alloy 1.19**,
deployed fully declaratively (Kustomize) into the `observability` namespace of
the local `orbstack` cluster.

Nothing is imperative: no `helm install`, no dashboard hand-created in the UI.
The entire desired state lives in this repository and `make deploy` reconciles it.

## Repository layout

```text
Makefile                              # Every command (make = help)
k8s/
├── kustomization.yaml                # Root Kustomize manifest
├── namespace.yaml                    # observability namespace
├── secret.yaml                       # Grafana admin credentials (GITIGNORED)
├── secret.example.yaml               # Secret template
├── prometheus/
│   ├── prometheus.yaml               # Scrape config (Kubernetes SD)
│   ├── rbac.yaml                     # SA + discovery ClusterRole
│   ├── secret.yaml                   # LibreChat scrape token (GITIGNORED)
│   ├── secret.example.yaml           # Token template
│   ├── pvc.yaml                      # Persistent TSDB (10Gi)
│   ├── deployment.yaml               # Prometheus v3.14.0
│   └── service.yaml                  # :9090
├── loki/
│   ├── loki.yaml                     # Monolithic config, filesystem storage
│   ├── pvc.yaml                      # Chunks + index (10Gi)
│   ├── deployment.yaml               # Loki 3.7.6 (-target=all)
│   └── service.yaml                  # :3100 (HTTP) / :9096 (gRPC)
├── tempo/
│   ├── tempo.yaml                    # Monolithic config + metrics-generator
│   ├── pvc.yaml                      # WAL + trace blocks (10Gi)
│   ├── deployment.yaml               # Tempo 3.0.3 (-target=all)
│   └── service.yaml                  # :3200, OTLP :4317/:4318, Zipkin :9411
├── alloy/
│   ├── config.alloy                  # Pod logs -> Loki pipeline
│   ├── rbac.yaml                     # SA + ClusterRole (including pods/log)
│   ├── daemonset.yaml                # Alloy v1.19.0 (one pod per node)
│   └── service.yaml                  # :12345 (UI + /metrics)
├── grafana/
│   ├── provisioning/
│   │   ├── datasources.yaml          # Datasources with stable UIDs
│   │   └── dashboard-providers.yaml  # File provider
│   ├── dashboards/
│   │   ├── kustomization.yaml        # ConfigMap WITHOUT hash (hot reload)
│   │   ├── obs-stack-health.json     # uid: obs-stack-health
│   │   ├── obs-keycloak.json         # uid: obs-keycloak
│   │   ├── obs-librechat.json        # uid: obs-librechat
│   │   └── obs-classic-models-api.json
│   ├── pvc.yaml                      # Grafana SQLite database (5Gi)
│   ├── deployment.yaml               # Grafana 13.2.0
│   └── service.yaml                  # :3000
└── ingress/
    ├── certificate.yaml              # cert-manager Certificate (sttlab-local-ca)
    └── ingress.yaml                  # Three NGINX Ingresses
```

## Getting started

```bash
make                     # Help: every available target
make deploy              # Deploy the whole stack
make hosts               # /etc/hosts entries to add
make grafana-password    # Admin credentials
make smoke               # End-to-end verification
```

| URL | Purpose |
|---|---|
| `https://grafana.sttlab.local` | Grafana UI |
| `https://prometheus.sttlab.local` | Prometheus UI (targets, expression browser) |
| `https://alloy.sttlab.local` | Alloy UI (log pipeline state) |

Sending traces from an in-cluster application:
`tempo.observability.svc.cluster.local:4317` (OTLP gRPC) or `:4318` (OTLP HTTP).

## Declarative dashboards with stable UIDs

This is the centrepiece of the repository: **a dashboard only ever exists as a
versioned JSON file**, never as an object created through the UI.

### Rules

1. **One `.json` file per dashboard**, under `k8s/grafana/dashboards/`.
2. **The `uid` field is stable and never changes.** It is what keeps URLs
   (`/d/<uid>/…`), cross-dashboard links, annotations and alerts alive across
   every redeployment. By convention the filename equals the UID
   (`obs-keycloak.json` -> `uid: obs-keycloak`).
3. **The `id` field must stay `null`** (or absent): it is an identifier internal
   to a Grafana instance and is not portable.
4. **Datasources are referenced by stable UID**, never by name:
   `"datasource": { "type": "prometheus", "uid": "prometheus" }`.
   Available UIDs: `prometheus`, `loki`, `tempo`.
5. `allowUiUpdates: false`: any change made in the UI is overwritten on the next
   reload. The file is the source of truth.

### Adding a dashboard

```bash
cp my-dashboard.json k8s/grafana/dashboards/obs-my-topic.json
# edit the JSON: "uid": "obs-my-topic", "id": null
make dashboards-sync     # update the list in dashboards/kustomization.yaml
make dashboards          # apply (without restarting Grafana)
```

`make dashboards-check` (run automatically by `dashboards` and `validate`)
verifies: valid JSON, `uid` present, `uid` unique, `id` null, file referenced in
the kustomization, filename/UID consistency.

### How the hot reload works

The `grafana-dashboards` ConfigMap is generated **without a name-suffix hash**
(`disableNameSuffixHash: true`), so `make dashboards` updates it *in place*: the
Grafana pod is not recreated and the file provider re-reads the directory
(`updateIntervalSeconds: 10`).

Measured end-to-end delay: **about 70 seconds** (kubelet ConfigMap volume sync
~60s + provider poll 10s). For immediate effect: `make restart-grafana`.

Conversely, the *provisioning* ConfigMaps (datasources, providers) **keep** their
hash suffix: those files are only read at startup, so the hash deliberately
forces a pod restart when they change.

## Persistence

| PVC | Component | Size | Contents | Retention |
|---|---|---|---|---|
| `prometheus-data` | Prometheus | 10Gi | TSDB | 15 d / 8 GB |
| `loki-data` | Loki | 10Gi | chunks + TSDB index | 168 h (compactor) |
| `tempo-data` | Tempo | 10Gi | WAL + trace blocks | 168 h |
| `grafana-data` | Grafana | 5Gi | SQLite database (users, prefs) | — |

> **Applies are server-side.** Client-side `kubectl apply` stores a full copy of
> each object in the `kubectl.kubernetes.io/last-applied-configuration`
> annotation, capped at 256KB; the dashboard ConfigMap passed that in aggregate
> at eight dashboards. `--server-side` tracks field ownership in metadata
> instead, which moves the ceiling to etcd's ~1MB per object. Splitting into one
> ConfigMap per dashboard is the next step if that is ever reached.

StorageClass `local-path` (the cluster default), `ReadWriteOnce`. The Deployments
use `strategy: Recreate`: with an RWO volume the old pod must release it before
the new one can start.

Alloy's read positions live on a `hostPath` (`/var/lib/alloy`), so restarting the
DaemonSet does not re-ingest every log.

### Wiping data

```bash
make pvc                             # PVC state
make flush-loki       CONFIRM=yes    # wipe the logs only
make flush-prometheus CONFIRM=yes    # wipe the TSDB only
make flush-pvc        CONFIRM=yes    # wipe EVERYTHING
```

The cycle is: `scale 0` -> wait for pod deletion -> `delete pvc` -> reapply the
kustomization (which recreates the PVC) -> `scale 1`. Every destructive target
requires `CONFIRM=yes`.

Dashboards are **not** affected by `flush-grafana`: they are reprovisioned from
the ConfigMaps at startup.

## Wiring the three signals

- **Prometheus** discovers its targets through the Kubernetes API: apiserver,
  kubelet, container resource metrics, plus any Service or Pod carrying the
  `prometheus.io/scrape: "true"` annotation (`prometheus.io/port`,
  `prometheus.io/path`).
- **Alloy** reads every pod's logs through the Kubernetes API (no `/var/log`
  mount) and pushes them to Loki with `namespace`, `pod`, `container`, `node`,
  `app`, `component` and `job` labels.
- **Tempo** receives OTLP, and its `metrics_generator` pushes span metrics and
  service graphs to Prometheus through its remote-write receiver
  (`--web.enable-remote-write-receiver`).
- **Grafana correlations**: Loki -> Tempo (`TraceID` derived field),
  Tempo -> Loki (`tracesToLogsV2`), Tempo -> Prometheus (`serviceMap`,
  `tracesToMetrics`), Prometheus exemplars -> Tempo.

### Container resource metrics

Prometheus scrapes the kubelet's **`/metrics/resource`** endpoint, not
`/metrics/cadvisor`. On this cluster the cadvisor endpoint answers with `machine_*`
series only and exposes no `container_*` at all, so CPU and memory panels would
stay empty. `/metrics/resource` — the source metrics-server itself reads —
provides `container_cpu_usage_seconds_total` and
`container_memory_working_set_bytes` with `namespace`/`pod`/`container` labels.

kube-state-metrics is not deployed, so container restarts are derived from
changes in `container_start_time_seconds`.

## Connected applications

| Application | Traces | Metrics | Logs | Dashboard |
|---|---|---|---|---|
| `keycloak` | OTel Java agent → `tempo:4318` | native `/metrics` (management port 9000) | JSON + MDC `trace_id` | `obs-keycloak` |
| `librechat` | OTel SDK → `tempo:4318` | built-in `/metrics`, Bearer token | stdout | `obs-librechat` |
| `classic-models-api` | `opentelemetry-instrument` → `tempo:4318` | django-prometheus `/metrics` | JSON + `otelTraceID` | `obs-classic-models-api` |

The six Kong gateways share a single dashboard, `obs-kong-gateways`, with a
`gateway` selector at the top (one, several, or all). Its variable resolves a
subtlety of this setup: the metrics label is the full gateway name
(`catalog-dev`) while the audit-log label is the short domain (`catalog`). The
variable holds the short form, PromQL re-adds the `-dev` suffix, and LogQL uses
it as-is — so one control drives both signals. The capture group in its regex
also drops the unrelated `kong` gateway, which carries no `-dev` suffix.

Their manifests live in the `infra-keycloak`, `infra-librechat` and
`classic-models-api` repositories; this repository only holds the platform.

Each application dashboard carries the same four sections: an overview of RED
indicators, HTTP traffic, a runtime section specific to the stack in use, and a
**Logs** row, followed by container resources.

### Distributed tracing

`obs-traces` searches traces with TraceQL, assembled from a service selector, a
minimum duration, a status, and a free-text field for conditions the selectors
do not cover. The status and free-text variables hold TraceQL *fragments* rather
than values, so an empty one contributes nothing and the surrounding query stays
valid — "no filter" is the default rather than a special case.

Opening a trace gives the waterfall, every span's attributes and events, and
from a span its logs and its RED metrics. The rest of the dashboard is the
aggregate view: rate, errors and latency percentiles from the span metrics
Tempo's own `metrics_generator` derives from the traces, the slowest and most
failing operations by span name, and the service-to-service edges Tempo infers.

Span name rather than endpoint is the unit worth looking at: a slow database
statement surfaces next to a slow HTTP handler, which is what tracing buys over
plain HTTP metrics.

The interactive node graph stays in Explore — `serviceMap` is a query type
Grafana's frontend assembles and it returns nothing through the dashboard query
API, so the dashboard shows the same edges as a table instead.

#### Span-to-logs correlation

The Tempo datasource resolves a span's logs with a custom query on
`${__trace.traceId}` rather than by mapping `service.name` to a Loki label. The
label mapping it replaces resolved to `app`, which Alloy derives from
`app.kubernetes.io/name`: that exists for Keycloak and the Classic Models API
but not for the six Kong gateways — and the gateways were the only place the
trace id actually appeared. The mapping pointed at the services where it could
not work.

Coverage today, each verified by taking a trace id and searching for it:

| Service | Trace id in logs | How |
|---|---|---|
| Kong gateways (×6) | yes | `x-trace-id` in the http-log record |
| Keycloak | yes | MDC, via `KC_LOG_MDC_ENABLED` |
| classic-models-api | yes | inbound `traceparent` in the gunicorn access line |
| LibreChat | no | would need the upstream image patched |

`tracesToMetrics` likewise had no queries defined and rendered a bare datasource
link; it now carries rate, error-rate and p95 queries against the span metrics,
scoped to the span's own service.

### Log dashboards, split by kind

Logs are searched from three dashboards rather than one, because the three
questions have nothing in common and a single view compromised on all of them.

| Dashboard | Question | Sources |
|---|---|---|
| `obs-access-logs` | Which HTTP requests came in? | Kong records, Keycloak access logger, Django gunicorn, ingress-nginx |
| `obs-application-logs` | What is the code reporting, and what broke? | Keycloak, LibreChat, Django, the Kong dataplanes' nginx error log |
| `obs-audit-logs` | Who authenticated, what changed, who called what? | Keycloak `org.keycloak.events`, Kong http-log records |

Each source keeps its own parser and they all emit the same field names, so the
frames merge into one table with consistent columns.

Keycloak's identity events are read with one independent `regexp` per field
rather than `line_format` piped into `logfmt`. Two reasons, both measured: that
combination is pathological on this data — each stage answers in 0.3s alone,
chained they time out at 60s and the table never renders — and the field set
differs per event type, so a single regexp spanning them would fail whole rows
on a missing field. Independent stages leave the absent field empty and keep the
row, and the query returns in 0.1s. Access logs converge on
method / path / status / client; application logs on level / message.

Three things this split fixes that a single consolidated view could not:

**The kinds are subtracted, not just filtered.** Application logs exclude
Keycloak's `access-log` and `events` loggers by name, the Django container's
NCSA lines by shape, and the Kong containers' stray access lines by requiring
the nginx date prefix. Without that, access lines drown the signal.

**Levels are read from the line, not inferred.** Loki's `detected_level` marks
78% of the Kong gateway output `unknown` even though every one of those lines
states `[notice]` or `[warn]`. Each parser extracts the level and lowercases it,
so nginx's `notice`, Keycloak's `INFO` and LibreChat's `info` land on one scale
— and the level selector has no `unknown` entry, because a line reaching the
table always states one.

**The scope selector actually scopes.** Each query carries both
`job=~"${source:regex}"` and a literal job matcher. With a single selector,
choosing "All" widened the query back to every stream in the cluster — which is
how MongoDB and cert-manager ended up in an application view.

Latency is populated for Kong only: the NCSA format the other sources use
carries no timing field. That is a property of the format, not a gap in the
query.

The NCSA sources are read with `regexp`, not `pattern`. A trailing `pattern`
capture consumes the rest of the line, so a `<size>` at the end swallowed the
referer and user-agent along with the byte count. Bounding the capture on `\d+`
stops it at the number, and `regexp` tolerates both shapes — Keycloak's access
line ends at the size, gunicorn's and nginx's carry more fields after it.

In every table the raw record is the last column, and the `organize`
transformation's `indexByName` keys on the original field names rather than the
renamed labels — keying it on the display names silently leaves the column order
to whatever order the frames arrived in.

### Parsed log tables### Parsed log tables

Every dashboard shows its logs twice: a **table** with one column per parsed
field, and, in a collapsed row underneath, the original **logs panel**.

The table is what makes a JSON record readable — status, method, route, latency
as sortable, filterable columns rather than a wall of braces. Its last column
holds the unparsed record, and every cell has the inspector enabled, so the raw
line is one click away.

The logs panel stays because a table cannot do what it does: render ANSI colour,
expand a record inline, tail live, and turn a `trace_id` into a link to Tempo.
It is collapsed so it costs nothing until needed.

Parsing is per-format, because these five sources agree on nothing:

| Source | Format | Parser |
|---|---|---|
| Kong audit | nested JSON | `\| json` with explicit field paths |
| Keycloak | JSON | `\| json` |
| classic-models-api | NCSA access lines | `\| pattern` |
| LibreChat | ANSI-coloured text | `line_format` to strip ANSI, then `\| pattern` |
| Observability stack | logfmt | `\| logfmt` with named fields |

Two choices worth knowing. The Kong and stack parsers name their fields
explicitly rather than extracting everything: a bare `\| json` on a Kong record
yields about seventy columns, and a bare `\| logfmt` explodes Loki's own
query-metrics lines. And `\| pattern` keeps lines it cannot match, with empty
columns, rather than dropping them — so the Django table never silently hides
an application log among the access lines.

The Logs row leans on Loki's `detected_level`, which is inferred at ingestion
time, so a single query works across all three log formats — Keycloak emits JSON,
LibreChat colour-coded text, and the Django API plain gunicorn access lines. No
per-application parser is needed. The volume panel is scoped to the application
container, while the two error panels cover the whole namespace so the
datastore's own failures surface alongside the application's. Where a log line
carries a `trace_id`, the Loki datasource's derived field turns it into a link
straight to the matching trace in Tempo.

### Connecting an application

1. **Traces** — point OTLP at Tempo:
   `OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.observability.svc.cluster.local:4318`
   (`:4317` for gRPC). RED span metrics and the service graph are then derived
   automatically by Tempo's `metrics_generator`, with no extra instrumentation.
2. **Metrics** — annotate the **pod** (not the Service):
   ```yaml
   annotations:
     prometheus.io/scrape: "true"
     prometheus.io/port: "9000"
     prometheus.io/path: "/metrics"
   ```
3. **Logs** — nothing to do: Alloy collects every pod in the cluster. For Grafana
   to jump from a log line to its trace, the line only needs to contain a
   `trace_id` (the Loki datasource's derived field).

### Token-protected endpoints

Annotation-based discovery cannot carry credentials, so an authenticated target
gets its own `scrape_config` in `k8s/prometheus/prometheus.yaml`, with the token
mounted from a Secret — see the `librechat` job, which reads
`/etc/prometheus/secrets/librechat/token`.

That token must be **identical** on both sides: `METRICS_SECRET` in the
`librechat-secrets` secret (namespace `librechat`) and the `token` key of the
`librechat-metrics` secret (namespace `observability`, file
`k8s/prometheus/secret.yaml`, gitignored).

A Kubernetes Secret is namespaced and a pod can only mount ones from its own
namespace, so the same value has to exist twice — once for LibreChat, which
**verifies** the token, once for Prometheus, which **presents** it. The risk is
silent drift: regenerating one side without the other sends the scrape back to
401 without breaking anything else. `make check-metrics-token` (included in
`make diag`) compares both copies.

## Diagnostics

```bash
make status              # pods, services, PVCs, ingresses
make diag                # + endpoints, certificate, failing containers, events
make smoke               # health of all five components, datasources, dashboards, targets
make targets             # Prometheus targets and their last scrape error
make logs-loki           # follow one component's logs
make top                 # CPU/memory usage
make pf-grafana          # port-forward when the ingress is unavailable
make diff                # difference between the repository and the cluster
make validate            # kustomize build + server dry-run + dashboard checks
```

## Secrets

`k8s/secret.yaml` and `k8s/prometheus/secret.yaml` are **gitignored**.
`make secret` generates the first from `secret.example.yaml` with a random admin
password if it does not exist (and is called automatically by `make deploy`).

The TLS certificate is not versioned either: it is issued by the existing
`sttlab-local-ca` cert-manager ClusterIssuer (`k8s/ingress/certificate.yaml`).

## Coexisting with what was already there

The `observability` namespace already hosts two deployments **not managed by this
repository**, left untouched:

- `jaeger` (installed by Helm) — it also exposes OTLP on `4317`/`4318`, but on its
  own Service, so there is no conflict with Tempo. Since all three applications
  now point at Tempo, Jaeger no longer receives anything; uninstall the Helm
  release whenever you want a single trace backend.
- `audit-log-collector` (Fluent Bit) — writes to its own PVC.

`make undeploy` and `make destroy` only run `kubectl delete -k k8s/`, so they
touch nothing but the resources described here.