Raw lines are close to unreadable when the payload is JSON, and the Kong audit
records are the worst case: a wall of braces where the interesting fields are
status, route and latency.
Each dashboard now leads with a table carrying one column per parsed field,
sortable and filterable, with the unparsed record as the last column and the
cell inspector enabled on every cell. The original logs panel moves into a
collapsed row underneath rather than disappearing: a table cannot render ANSI
colour, expand a record inline, tail live, or turn a trace id into a link to
Tempo.
Parsing is per-format since the five sources agree on nothing — nested JSON for
Kong, JSON for Keycloak, NCSA access lines for the Django API, ANSI-coloured
text for LibreChat, logfmt for the stack itself.
The Kong and stack parsers name their fields explicitly rather than extracting
everything: a bare `| json` on a Kong record yields around seventy columns, and
a bare `| logfmt` explodes Loki's own query-metrics lines into dozens. Columns
are selected by include-pattern rather than by excluding names, because the
stream labels that ride along are not a fixed set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploys the whole stack into the observability namespace of the local
cluster with Kustomize alone — no helm install, no dashboard created by
hand in the UI. The Makefile carries every command: deploy, dashboards,
diagnostics, PVC flush, teardown.
Dashboards are files, never UI objects. Each carries a stable uid, which
is what keeps URLs, cross-links and alerts alive across redeployments, and
references datasources by stable uid rather than by name. The dashboard
ConfigMap is generated without a name-suffix hash so applying it updates
it in place and Grafana reloads without a restart, while the provisioning
ConfigMaps keep their hash precisely so that a change there does force one.
Prometheus scrapes the kubelet /metrics/resource endpoint rather than
/metrics/cadvisor: on this cluster cadvisor exposes machine_* series only
and no container_* at all, which would leave every CPU and memory panel
empty.
Alloy ships every pod's logs to Loki through the Kubernetes API, so no
/var/log mount is needed. Tempo's metrics_generator feeds span metrics and
the service graph back into Prometheus, and the datasources are wired so a
log line's trace_id opens the matching trace.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland