The dashboard reimplemented, less well, what Explore already does: its search
panel had no TraceQL query builder and no autocompletion, and its service
dependency table replaced an interactive node graph with a list. Searching and
exploring traces belongs in Explore.
The datasource correlations this work also fixed are kept — they are what was
actually missing, and they apply in Explore exactly as they would in a
dashboard.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Trace search lived only in Explore. `obs-traces` brings it into a dashboard:
TraceQL assembled from a service selector, a minimum duration, a status and a
free-text field, plus the aggregate RED view from the span metrics Tempo derives
from the traces themselves.
The status and free-text variables hold TraceQL fragments rather than values, so
an empty one contributes nothing and the query stays valid — the same reason the
log dashboards inject their search term the way they do.
Both span correlations were declared but non-functional.
tracesToLogsV2 mapped service.name onto Loki's `app` label, which Alloy derives
from app.kubernetes.io/name. That label exists for Keycloak and the Classic
Models API but not for the six Kong gateways or LibreChat — and the gateways are
the only place the trace id actually appears in the logs, so the mapping pointed
squarely at the services where it could not resolve. Replaced with a custom
query on ${__trace.traceId}, which matches any source writing the id anywhere in
the line.
tracesToMetrics had no queries at all and rendered a bare datasource link. It
now carries rate, error-rate and p95 against traces_spanmetrics_*, scoped to the
span's own service.
The service graph stays a table: serviceMap is a query type Grafana's frontend
assembles for Explore and returns no frames through the dashboard query API.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The panel never displayed: its query combined `line_format` with `logfmt`, and
on this data that pairing is pathological. Measured on a 24h range, the stream
selector with its json filter answers in 0.7s, adding line_format alone 0.3s,
adding logfmt alone 0.2s — chained, 60s and a timeout. The bar panels above it
survived because an aggregation streams; the log query did not.
Replaced with one independent `regexp` per field, reading the key="value" pairs
straight out of the JSON message without reformatting the line. 0.1s, and the
raw record now stays raw since nothing rewrites it.
Independent stages also fix a correctness problem the single-regexp version
would have had: the field set differs per event type — LOGIN_ERROR carries
`error` and `grant_type`, USER_SESSION_DELETED carries neither — so one regexp
spanning them all would drop whole rows on a missing field. All four types
observed on this cluster now extract correctly.
Failed and successful logins widen from LOGIN_ERROR and LOGIN to the *_ERROR
and CODE_TO_TOKEN families: a failed refresh or code exchange is the same
signal, and TOKEN_EXCHANGE dominates this cluster's traffic.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three defects, all mine.
Column order was ignored because organize's indexByName was keyed on the
renamed display labels. Grafana keys it on the original field names, so the
mapping matched nothing and the order fell back to whatever order the frames
arrived in — which is why the raw line landed second. Every table is affected,
not just the access one.
The byte count was wrong for the NCSA sources: a trailing `pattern` capture
takes the rest of the line, so `<size>` held `16 "-" "kube-probe/1.35"` rather
than 16, and for nginx it swallowed the whole referer and user-agent. Replaced
with a `regexp` parser bounding the capture on \d+, which also handles both
line shapes — Keycloak's access line ends at the size while gunicorn's and
nginx's carry more fields after it.
Access log columns are now time, status, method, path, source, client, latency,
bytes, raw line, with the raw record last in every table.
gunicorn and nginx now share one query since their format is identical, which
also halves the duplicate class names the status panel showed — one series per
query returning the same class.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One consolidated search compromised on all three questions at once. It also
had a real defect: the Source selector carried `.+` as its All value, so
choosing "All" widened the query back to every stream in the cluster and the
scope preset only ever populated the picker. MongoDB and cert-manager showed up
in what was meant to be an application view.
Each query now carries both `job=~"${source:regex}"` and a literal job matcher,
so the selector actually restricts rather than widens.
The three kinds are subtracted from each other rather than merely filtered:
application logs exclude Keycloak's access-log and events loggers by name, the
Django container's NCSA lines by shape, and the Kong containers' stray access
lines by requiring the nginx date prefix.
Levels are read from the line rather than taken from Loki's detected_level.
That inference marks 78% of the Kong gateway output `unknown` even though every
one of those lines states [notice] or [warn], so filtering on it hid most of
the gateway's own logs. Each parser extracts the level and lowercases it, which
puts nginx's notice, Keycloak's INFO and LibreChat's info on one scale — and
removes the need for an `unknown` entry in the selector.
Applies move to server-side. Client-side apply stores a full copy of each
object in the last-applied-configuration annotation, capped at 256KB, and the
dashboard ConfigMap passed that at eight dashboards; the failure surfaced as
"metadata.annotations: Too long". Server-side apply tracks field ownership in
metadata instead, moving the ceiling to etcd's ~1MB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The per-application dashboards answer "how is this component behaving". This
one answers "where did that string come from", across every source at once, and
is kept separate so neither has to compromise for the other.
Scope is a curated preset rather than a free namespace picker: the cluster
carries 48 log sources and most are infrastructure noise. The presets are job
regexes over the <namespace>/<container> label Alloy sets — except kong-audit,
which Fluent Bit ships with no namespace at all. That mismatch between the two
collection paths is the reason a single naive selector cannot span both, and
why the presets spell the sources out.
The search term is injected inside LogQL backticks, a raw string, so a quote
typed in the box cannot break the query; an empty box yields `(?i)`, a valid
regex matching every line, which makes "no search" the default rather than a
special case. The four stat panels carry `or vector(0)` so an empty result
reads as 0 rather than "No data" — on a search box, finding nothing is an
answer, not a failure.
Two deliberate omissions. No per-format parsing: the Message column holds the
line as emitted, because the five formats agree on nothing and that is what the
per-application dashboards are for. And the level filter is partial by nature —
roughly 40% of lines are `unknown`, since access lines and Kong records carry
no severity for Loki to infer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Raw lines are close to unreadable when the payload is JSON, and the Kong audit
records are the worst case: a wall of braces where the interesting fields are
status, route and latency.
Each dashboard now leads with a table carrying one column per parsed field,
sortable and filterable, with the unparsed record as the last column and the
cell inspector enabled on every cell. The original logs panel moves into a
collapsed row underneath rather than disappearing: a table cannot render ANSI
colour, expand a record inline, tail live, or turn a trace id into a link to
Tempo.
Parsing is per-format since the five sources agree on nothing — nested JSON for
Kong, JSON for Keycloak, NCSA access lines for the Django API, ANSI-coloured
text for LibreChat, logfmt for the stack itself.
The Kong and stack parsers name their fields explicitly rather than extracting
everything: a bare `| json` on a Kong record yields around seventy columns, and
a bare `| logfmt` explodes Loki's own query-metrics lines into dozens. Columns
are selected by include-pattern rather than by excluding names, because the
stream labels that ride along are not a fixed set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One dashboard for all six gateways, with a selector accepting one, several or
all of them.
The selector has to bridge a mismatch: metrics carry the full gateway name
(catalog-dev) while audit logs carry the short domain (catalog), because the
former comes from the scrape job's label and the latter from the URI path each
gateway posts its http-log to. The variable holds the short form, PromQL
re-adds the -dev suffix and LogQL uses it as-is, so a single control drives
both signals. The capture group in the variable's regex also excludes the
unrelated `kong` gateway, which has no -dev suffix.
Latency is split three ways rather than shown as one number: total request
latency, time spent inside Kong, and upstream latency. The gap between the
first and the last is what the gateway itself costs, which is the question
worth asking of a proxy.
Status classes come from label_replace on the `code` label for metrics and
from LogQL's json parser for the audit records, since those carry no severity
of their own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploys the whole stack into the observability namespace of the local
cluster with Kustomize alone — no helm install, no dashboard created by
hand in the UI. The Makefile carries every command: deploy, dashboards,
diagnostics, PVC flush, teardown.
Dashboards are files, never UI objects. Each carries a stable uid, which
is what keeps URLs, cross-links and alerts alive across redeployments, and
references datasources by stable uid rather than by name. The dashboard
ConfigMap is generated without a name-suffix hash so applying it updates
it in place and Grafana reloads without a restart, while the provisioning
ConfigMaps keep their hash precisely so that a change there does force one.
Prometheus scrapes the kubelet /metrics/resource endpoint rather than
/metrics/cadvisor: on this cluster cadvisor exposes machine_* series only
and no container_* at all, which would leave every CPU and memory panel
empty.
Alloy ships every pod's logs to Loki through the Kubernetes API, so no
/var/log mount is needed. Tempo's metrics_generator feeds span metrics and
the service graph back into Prometheus, and the datasources are wired so a
log line's trace_id opens the matching trace.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland