Stéphane Tailland
40ffac4acf
The per-application dashboards answer "how is this component behaving". This one answers "where did that string come from", across every source at once, and is kept separate so neither has to compromise for the other. Scope is a curated preset rather than a free namespace picker: the cluster carries 48 log sources and most are infrastructure noise. The presets are job regexes over the <namespace>/<container> label Alloy sets — except kong-audit, which Fluent Bit ships with no namespace at all. That mismatch between the two collection paths is the reason a single naive selector cannot span both, and why the presets spell the sources out. The search term is injected inside LogQL backticks, a raw string, so a quote typed in the box cannot break the query; an empty box yields `(?i)`, a valid regex matching every line, which makes "no search" the default rather than a special case. The four stat panels carry `or vector(0)` so an empty result reads as 0 rather than "No data" — on a search box, finding nothing is an answer, not a failure. Two deliberate omissions. No per-format parsing: the Message column holds the line as emitted, because the five formats agree on nothing and that is what the per-application dashboards are for. And the level filter is partial by nature — roughly 40% of lines are `unknown`, since access lines and Kong records carry no severity for Loki to infer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
347 lines
17 KiB
Markdown
347 lines
17 KiB
Markdown
# Infra Observability (Kubernetes / declarative)
|
|
|
|
**Grafana 13.2 / Prometheus 3.14 / Loki 3.7 / Tempo 3.0** plus **Alloy 1.19**,
|
|
deployed fully declaratively (Kustomize) into the `observability` namespace of
|
|
the local `orbstack` cluster.
|
|
|
|
Nothing is imperative: no `helm install`, no dashboard hand-created in the UI.
|
|
The entire desired state lives in this repository and `make deploy` reconciles it.
|
|
|
|
## Repository layout
|
|
|
|
```text
|
|
Makefile # Every command (make = help)
|
|
k8s/
|
|
├── kustomization.yaml # Root Kustomize manifest
|
|
├── namespace.yaml # observability namespace
|
|
├── secret.yaml # Grafana admin credentials (GITIGNORED)
|
|
├── secret.example.yaml # Secret template
|
|
├── prometheus/
|
|
│ ├── prometheus.yaml # Scrape config (Kubernetes SD)
|
|
│ ├── rbac.yaml # SA + discovery ClusterRole
|
|
│ ├── secret.yaml # LibreChat scrape token (GITIGNORED)
|
|
│ ├── secret.example.yaml # Token template
|
|
│ ├── pvc.yaml # Persistent TSDB (10Gi)
|
|
│ ├── deployment.yaml # Prometheus v3.14.0
|
|
│ └── service.yaml # :9090
|
|
├── loki/
|
|
│ ├── loki.yaml # Monolithic config, filesystem storage
|
|
│ ├── pvc.yaml # Chunks + index (10Gi)
|
|
│ ├── deployment.yaml # Loki 3.7.6 (-target=all)
|
|
│ └── service.yaml # :3100 (HTTP) / :9096 (gRPC)
|
|
├── tempo/
|
|
│ ├── tempo.yaml # Monolithic config + metrics-generator
|
|
│ ├── pvc.yaml # WAL + trace blocks (10Gi)
|
|
│ ├── deployment.yaml # Tempo 3.0.3 (-target=all)
|
|
│ └── service.yaml # :3200, OTLP :4317/:4318, Zipkin :9411
|
|
├── alloy/
|
|
│ ├── config.alloy # Pod logs -> Loki pipeline
|
|
│ ├── rbac.yaml # SA + ClusterRole (including pods/log)
|
|
│ ├── daemonset.yaml # Alloy v1.19.0 (one pod per node)
|
|
│ └── service.yaml # :12345 (UI + /metrics)
|
|
├── grafana/
|
|
│ ├── provisioning/
|
|
│ │ ├── datasources.yaml # Datasources with stable UIDs
|
|
│ │ └── dashboard-providers.yaml # File provider
|
|
│ ├── dashboards/
|
|
│ │ ├── kustomization.yaml # ConfigMap WITHOUT hash (hot reload)
|
|
│ │ ├── obs-stack-health.json # uid: obs-stack-health
|
|
│ │ ├── obs-keycloak.json # uid: obs-keycloak
|
|
│ │ ├── obs-librechat.json # uid: obs-librechat
|
|
│ │ └── obs-classic-models-api.json
|
|
│ ├── pvc.yaml # Grafana SQLite database (5Gi)
|
|
│ ├── deployment.yaml # Grafana 13.2.0
|
|
│ └── service.yaml # :3000
|
|
└── ingress/
|
|
├── certificate.yaml # cert-manager Certificate (sttlab-local-ca)
|
|
└── ingress.yaml # Three NGINX Ingresses
|
|
```
|
|
|
|
## Getting started
|
|
|
|
```bash
|
|
make # Help: every available target
|
|
make deploy # Deploy the whole stack
|
|
make hosts # /etc/hosts entries to add
|
|
make grafana-password # Admin credentials
|
|
make smoke # End-to-end verification
|
|
```
|
|
|
|
| URL | Purpose |
|
|
|---|---|
|
|
| `https://grafana.sttlab.local` | Grafana UI |
|
|
| `https://prometheus.sttlab.local` | Prometheus UI (targets, expression browser) |
|
|
| `https://alloy.sttlab.local` | Alloy UI (log pipeline state) |
|
|
|
|
Sending traces from an in-cluster application:
|
|
`tempo.observability.svc.cluster.local:4317` (OTLP gRPC) or `:4318` (OTLP HTTP).
|
|
|
|
## Declarative dashboards with stable UIDs
|
|
|
|
This is the centrepiece of the repository: **a dashboard only ever exists as a
|
|
versioned JSON file**, never as an object created through the UI.
|
|
|
|
### Rules
|
|
|
|
1. **One `.json` file per dashboard**, under `k8s/grafana/dashboards/`.
|
|
2. **The `uid` field is stable and never changes.** It is what keeps URLs
|
|
(`/d/<uid>/…`), cross-dashboard links, annotations and alerts alive across
|
|
every redeployment. By convention the filename equals the UID
|
|
(`obs-keycloak.json` -> `uid: obs-keycloak`).
|
|
3. **The `id` field must stay `null`** (or absent): it is an identifier internal
|
|
to a Grafana instance and is not portable.
|
|
4. **Datasources are referenced by stable UID**, never by name:
|
|
`"datasource": { "type": "prometheus", "uid": "prometheus" }`.
|
|
Available UIDs: `prometheus`, `loki`, `tempo`.
|
|
5. `allowUiUpdates: false`: any change made in the UI is overwritten on the next
|
|
reload. The file is the source of truth.
|
|
|
|
### Adding a dashboard
|
|
|
|
```bash
|
|
cp my-dashboard.json k8s/grafana/dashboards/obs-my-topic.json
|
|
# edit the JSON: "uid": "obs-my-topic", "id": null
|
|
make dashboards-sync # update the list in dashboards/kustomization.yaml
|
|
make dashboards # apply (without restarting Grafana)
|
|
```
|
|
|
|
`make dashboards-check` (run automatically by `dashboards` and `validate`)
|
|
verifies: valid JSON, `uid` present, `uid` unique, `id` null, file referenced in
|
|
the kustomization, filename/UID consistency.
|
|
|
|
### How the hot reload works
|
|
|
|
The `grafana-dashboards` ConfigMap is generated **without a name-suffix hash**
|
|
(`disableNameSuffixHash: true`), so `make dashboards` updates it *in place*: the
|
|
Grafana pod is not recreated and the file provider re-reads the directory
|
|
(`updateIntervalSeconds: 10`).
|
|
|
|
Measured end-to-end delay: **about 70 seconds** (kubelet ConfigMap volume sync
|
|
~60s + provider poll 10s). For immediate effect: `make restart-grafana`.
|
|
|
|
Conversely, the *provisioning* ConfigMaps (datasources, providers) **keep** their
|
|
hash suffix: those files are only read at startup, so the hash deliberately
|
|
forces a pod restart when they change.
|
|
|
|
## Persistence
|
|
|
|
| PVC | Component | Size | Contents | Retention |
|
|
|---|---|---|---|---|
|
|
| `prometheus-data` | Prometheus | 10Gi | TSDB | 15 d / 8 GB |
|
|
| `loki-data` | Loki | 10Gi | chunks + TSDB index | 168 h (compactor) |
|
|
| `tempo-data` | Tempo | 10Gi | WAL + trace blocks | 168 h |
|
|
| `grafana-data` | Grafana | 5Gi | SQLite database (users, prefs) | — |
|
|
|
|
StorageClass `local-path` (the cluster default), `ReadWriteOnce`. The Deployments
|
|
use `strategy: Recreate`: with an RWO volume the old pod must release it before
|
|
the new one can start.
|
|
|
|
Alloy's read positions live on a `hostPath` (`/var/lib/alloy`), so restarting the
|
|
DaemonSet does not re-ingest every log.
|
|
|
|
### Wiping data
|
|
|
|
```bash
|
|
make pvc # PVC state
|
|
make flush-loki CONFIRM=yes # wipe the logs only
|
|
make flush-prometheus CONFIRM=yes # wipe the TSDB only
|
|
make flush-pvc CONFIRM=yes # wipe EVERYTHING
|
|
```
|
|
|
|
The cycle is: `scale 0` -> wait for pod deletion -> `delete pvc` -> reapply the
|
|
kustomization (which recreates the PVC) -> `scale 1`. Every destructive target
|
|
requires `CONFIRM=yes`.
|
|
|
|
Dashboards are **not** affected by `flush-grafana`: they are reprovisioned from
|
|
the ConfigMaps at startup.
|
|
|
|
## Wiring the three signals
|
|
|
|
- **Prometheus** discovers its targets through the Kubernetes API: apiserver,
|
|
kubelet, container resource metrics, plus any Service or Pod carrying the
|
|
`prometheus.io/scrape: "true"` annotation (`prometheus.io/port`,
|
|
`prometheus.io/path`).
|
|
- **Alloy** reads every pod's logs through the Kubernetes API (no `/var/log`
|
|
mount) and pushes them to Loki with `namespace`, `pod`, `container`, `node`,
|
|
`app`, `component` and `job` labels.
|
|
- **Tempo** receives OTLP, and its `metrics_generator` pushes span metrics and
|
|
service graphs to Prometheus through its remote-write receiver
|
|
(`--web.enable-remote-write-receiver`).
|
|
- **Grafana correlations**: Loki -> Tempo (`TraceID` derived field),
|
|
Tempo -> Loki (`tracesToLogsV2`), Tempo -> Prometheus (`serviceMap`,
|
|
`tracesToMetrics`), Prometheus exemplars -> Tempo.
|
|
|
|
### Container resource metrics
|
|
|
|
Prometheus scrapes the kubelet's **`/metrics/resource`** endpoint, not
|
|
`/metrics/cadvisor`. On this cluster the cadvisor endpoint answers with `machine_*`
|
|
series only and exposes no `container_*` at all, so CPU and memory panels would
|
|
stay empty. `/metrics/resource` — the source metrics-server itself reads —
|
|
provides `container_cpu_usage_seconds_total` and
|
|
`container_memory_working_set_bytes` with `namespace`/`pod`/`container` labels.
|
|
|
|
kube-state-metrics is not deployed, so container restarts are derived from
|
|
changes in `container_start_time_seconds`.
|
|
|
|
## Connected applications
|
|
|
|
| Application | Traces | Metrics | Logs | Dashboard |
|
|
|---|---|---|---|---|
|
|
| `keycloak` | OTel Java agent → `tempo:4318` | native `/metrics` (management port 9000) | JSON + MDC `trace_id` | `obs-keycloak` |
|
|
| `librechat` | OTel SDK → `tempo:4318` | built-in `/metrics`, Bearer token | stdout | `obs-librechat` |
|
|
| `classic-models-api` | `opentelemetry-instrument` → `tempo:4318` | django-prometheus `/metrics` | JSON + `otelTraceID` | `obs-classic-models-api` |
|
|
|
|
The six Kong gateways share a single dashboard, `obs-kong-gateways`, with a
|
|
`gateway` selector at the top (one, several, or all). Its variable resolves a
|
|
subtlety of this setup: the metrics label is the full gateway name
|
|
(`catalog-dev`) while the audit-log label is the short domain (`catalog`). The
|
|
variable holds the short form, PromQL re-adds the `-dev` suffix, and LogQL uses
|
|
it as-is — so one control drives both signals. The capture group in its regex
|
|
also drops the unrelated `kong` gateway, which carries no `-dev` suffix.
|
|
|
|
Their manifests live in the `infra-keycloak`, `infra-librechat` and
|
|
`classic-models-api` repositories; this repository only holds the platform.
|
|
|
|
Each application dashboard carries the same four sections: an overview of RED
|
|
indicators, HTTP traffic, a runtime section specific to the stack in use, and a
|
|
**Logs** row, followed by container resources.
|
|
|
|
### Consolidated log search
|
|
|
|
`obs-log-search` searches every source at once, and is deliberately separate
|
|
from the per-application dashboards: those answer "how is this component
|
|
behaving", this one answers "where did that string come from".
|
|
|
|
Four controls: a **scope** preset, a **source** picker chained to it, a
|
|
**level** filter, and a free-text **search** box. The search term is injected as
|
|
a case-insensitive line filter inside LogQL backticks — a raw string, so a quote
|
|
typed in the box cannot break the query, and an empty box yields `(?i)`, a valid
|
|
regex matching every line. That is what makes "no search" the natural default
|
|
rather than a special case.
|
|
|
|
Scope is a curated preset list rather than a free namespace picker, because the
|
|
cluster carries 48 log sources and most are infrastructure noise. The presets
|
|
are job regexes matched against the `<namespace>/<container>` label Alloy sets —
|
|
except `kong-audit`, which Fluent Bit ships with no namespace at all, and which
|
|
is why a single naive selector cannot span both collection paths.
|
|
|
|
Two things this dashboard deliberately does not do. It does not parse per
|
|
format: its Message column holds the line as emitted, since the five sources
|
|
agree on nothing and that is what the per-application dashboards are for. And
|
|
its level filter is honest about being partial — around 40% of lines are
|
|
`unknown` because access lines and Kong records carry no severity for Loki to
|
|
infer, so for those the text is the only thing worth searching.
|
|
|
|
### Parsed log tables
|
|
|
|
Every dashboard shows its logs twice: a **table** with one column per parsed
|
|
field, and, in a collapsed row underneath, the original **logs panel**.
|
|
|
|
The table is what makes a JSON record readable — status, method, route, latency
|
|
as sortable, filterable columns rather than a wall of braces. Its last column
|
|
holds the unparsed record, and every cell has the inspector enabled, so the raw
|
|
line is one click away.
|
|
|
|
The logs panel stays because a table cannot do what it does: render ANSI colour,
|
|
expand a record inline, tail live, and turn a `trace_id` into a link to Tempo.
|
|
It is collapsed so it costs nothing until needed.
|
|
|
|
Parsing is per-format, because these five sources agree on nothing:
|
|
|
|
| Source | Format | Parser |
|
|
|---|---|---|
|
|
| Kong audit | nested JSON | `\| json` with explicit field paths |
|
|
| Keycloak | JSON | `\| json` |
|
|
| classic-models-api | NCSA access lines | `\| pattern` |
|
|
| LibreChat | ANSI-coloured text | `line_format` to strip ANSI, then `\| pattern` |
|
|
| Observability stack | logfmt | `\| logfmt` with named fields |
|
|
|
|
Two choices worth knowing. The Kong and stack parsers name their fields
|
|
explicitly rather than extracting everything: a bare `\| json` on a Kong record
|
|
yields about seventy columns, and a bare `\| logfmt` explodes Loki's own
|
|
query-metrics lines. And `\| pattern` keeps lines it cannot match, with empty
|
|
columns, rather than dropping them — so the Django table never silently hides
|
|
an application log among the access lines.
|
|
|
|
The Logs row leans on Loki's `detected_level`, which is inferred at ingestion
|
|
time, so a single query works across all three log formats — Keycloak emits JSON,
|
|
LibreChat colour-coded text, and the Django API plain gunicorn access lines. No
|
|
per-application parser is needed. The volume panel is scoped to the application
|
|
container, while the two error panels cover the whole namespace so the
|
|
datastore's own failures surface alongside the application's. Where a log line
|
|
carries a `trace_id`, the Loki datasource's derived field turns it into a link
|
|
straight to the matching trace in Tempo.
|
|
|
|
### Connecting an application
|
|
|
|
1. **Traces** — point OTLP at Tempo:
|
|
`OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.observability.svc.cluster.local:4318`
|
|
(`:4317` for gRPC). RED span metrics and the service graph are then derived
|
|
automatically by Tempo's `metrics_generator`, with no extra instrumentation.
|
|
2. **Metrics** — annotate the **pod** (not the Service):
|
|
```yaml
|
|
annotations:
|
|
prometheus.io/scrape: "true"
|
|
prometheus.io/port: "9000"
|
|
prometheus.io/path: "/metrics"
|
|
```
|
|
3. **Logs** — nothing to do: Alloy collects every pod in the cluster. For Grafana
|
|
to jump from a log line to its trace, the line only needs to contain a
|
|
`trace_id` (the Loki datasource's derived field).
|
|
|
|
### Token-protected endpoints
|
|
|
|
Annotation-based discovery cannot carry credentials, so an authenticated target
|
|
gets its own `scrape_config` in `k8s/prometheus/prometheus.yaml`, with the token
|
|
mounted from a Secret — see the `librechat` job, which reads
|
|
`/etc/prometheus/secrets/librechat/token`.
|
|
|
|
That token must be **identical** on both sides: `METRICS_SECRET` in the
|
|
`librechat-secrets` secret (namespace `librechat`) and the `token` key of the
|
|
`librechat-metrics` secret (namespace `observability`, file
|
|
`k8s/prometheus/secret.yaml`, gitignored).
|
|
|
|
A Kubernetes Secret is namespaced and a pod can only mount ones from its own
|
|
namespace, so the same value has to exist twice — once for LibreChat, which
|
|
**verifies** the token, once for Prometheus, which **presents** it. The risk is
|
|
silent drift: regenerating one side without the other sends the scrape back to
|
|
401 without breaking anything else. `make check-metrics-token` (included in
|
|
`make diag`) compares both copies.
|
|
|
|
## Diagnostics
|
|
|
|
```bash
|
|
make status # pods, services, PVCs, ingresses
|
|
make diag # + endpoints, certificate, failing containers, events
|
|
make smoke # health of all five components, datasources, dashboards, targets
|
|
make targets # Prometheus targets and their last scrape error
|
|
make logs-loki # follow one component's logs
|
|
make top # CPU/memory usage
|
|
make pf-grafana # port-forward when the ingress is unavailable
|
|
make diff # difference between the repository and the cluster
|
|
make validate # kustomize build + server dry-run + dashboard checks
|
|
```
|
|
|
|
## Secrets
|
|
|
|
`k8s/secret.yaml` and `k8s/prometheus/secret.yaml` are **gitignored**.
|
|
`make secret` generates the first from `secret.example.yaml` with a random admin
|
|
password if it does not exist (and is called automatically by `make deploy`).
|
|
|
|
The TLS certificate is not versioned either: it is issued by the existing
|
|
`sttlab-local-ca` cert-manager ClusterIssuer (`k8s/ingress/certificate.yaml`).
|
|
|
|
## Coexisting with what was already there
|
|
|
|
The `observability` namespace already hosts two deployments **not managed by this
|
|
repository**, left untouched:
|
|
|
|
- `jaeger` (installed by Helm) — it also exposes OTLP on `4317`/`4318`, but on its
|
|
own Service, so there is no conflict with Tempo. Since all three applications
|
|
now point at Tempo, Jaeger no longer receives anything; uninstall the Helm
|
|
release whenever you want a single trace backend.
|
|
- `audit-log-collector` (Fluent Bit) — writes to its own PVC.
|
|
|
|
`make undeploy` and `make destroy` only run `kubectl delete -k k8s/`, so they
|
|
touch nothing but the resources described here.
|