Explore's unfiltered trace search is unusable on this cluster: Tempo does
not return {} newest-first, so raising the limit does not walk backwards
in time and the health-check traffic — 91 Keycloak token calls and dozens
of /health/ and /metrics probes an hour — crowds real transactions out of
the first 20 results. Measured: at a fixed instant, limit=20 returned
06:50:43-06:51:07 while limit=100 returned 06:47:44-06:50:02.
Four panels:
* the traces on the dashboard's time range, filtered by service, each
row linking its trace id into the traceId variable;
* the HTTP hops between components — the SERVER-kind spans, one per
service entry, with method/status/route projected via select(), so
the Express middleware, DNS and SQL spans stay out;
* the Kong plugin spans with their individual durations;
* the Loki lines carrying the selected trace id.
No metric panels, on purpose. A span's own duration is already in the
span tables and is measured rather than estimated; aggregate RED by span
name is what obs-classic-models-api, obs-kong-gateways and obs-librechat
already show. Neither answers "the metrics of this span", so putting them
here only duplicated existing dashboards.
Two correlation limits are written into the panel descriptions rather
than left to be discovered: the join is by trace id and not span id
because no emitter logs a span id, and LibreChat logs carry no trace id
at all so its lines never appear.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dashboard reimplemented, less well, what Explore already does: its search
panel had no TraceQL query builder and no autocompletion, and its service
dependency table replaced an interactive node graph with a list. Searching and
exploring traces belongs in Explore.
The datasource correlations this work also fixed are kept — they are what was
actually missing, and they apply in Explore exactly as they would in a
dashboard.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Trace search lived only in Explore. `obs-traces` brings it into a dashboard:
TraceQL assembled from a service selector, a minimum duration, a status and a
free-text field, plus the aggregate RED view from the span metrics Tempo derives
from the traces themselves.
The status and free-text variables hold TraceQL fragments rather than values, so
an empty one contributes nothing and the query stays valid — the same reason the
log dashboards inject their search term the way they do.
Both span correlations were declared but non-functional.
tracesToLogsV2 mapped service.name onto Loki's `app` label, which Alloy derives
from app.kubernetes.io/name. That label exists for Keycloak and the Classic
Models API but not for the six Kong gateways or LibreChat — and the gateways are
the only place the trace id actually appears in the logs, so the mapping pointed
squarely at the services where it could not resolve. Replaced with a custom
query on ${__trace.traceId}, which matches any source writing the id anywhere in
the line.
tracesToMetrics had no queries at all and rendered a bare datasource link. It
now carries rate, error-rate and p95 against traces_spanmetrics_*, scoped to the
span's own service.
The service graph stays a table: serviceMap is a query type Grafana's frontend
assembles for Explore and returns no frames through the dashboard query API.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One consolidated search compromised on all three questions at once. It also
had a real defect: the Source selector carried `.+` as its All value, so
choosing "All" widened the query back to every stream in the cluster and the
scope preset only ever populated the picker. MongoDB and cert-manager showed up
in what was meant to be an application view.
Each query now carries both `job=~"${source:regex}"` and a literal job matcher,
so the selector actually restricts rather than widens.
The three kinds are subtracted from each other rather than merely filtered:
application logs exclude Keycloak's access-log and events loggers by name, the
Django container's NCSA lines by shape, and the Kong containers' stray access
lines by requiring the nginx date prefix.
Levels are read from the line rather than taken from Loki's detected_level.
That inference marks 78% of the Kong gateway output `unknown` even though every
one of those lines states [notice] or [warn], so filtering on it hid most of
the gateway's own logs. Each parser extracts the level and lowercases it, which
puts nginx's notice, Keycloak's INFO and LibreChat's info on one scale — and
removes the need for an `unknown` entry in the selector.
Applies move to server-side. Client-side apply stores a full copy of each
object in the last-applied-configuration annotation, capped at 256KB, and the
dashboard ConfigMap passed that at eight dashboards; the failure surfaced as
"metadata.annotations: Too long". Server-side apply tracks field ownership in
metadata instead, moving the ceiling to etcd's ~1MB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The per-application dashboards answer "how is this component behaving". This
one answers "where did that string come from", across every source at once, and
is kept separate so neither has to compromise for the other.
Scope is a curated preset rather than a free namespace picker: the cluster
carries 48 log sources and most are infrastructure noise. The presets are job
regexes over the <namespace>/<container> label Alloy sets — except kong-audit,
which Fluent Bit ships with no namespace at all. That mismatch between the two
collection paths is the reason a single naive selector cannot span both, and
why the presets spell the sources out.
The search term is injected inside LogQL backticks, a raw string, so a quote
typed in the box cannot break the query; an empty box yields `(?i)`, a valid
regex matching every line, which makes "no search" the default rather than a
special case. The four stat panels carry `or vector(0)` so an empty result
reads as 0 rather than "No data" — on a search box, finding nothing is an
answer, not a failure.
Two deliberate omissions. No per-format parsing: the Message column holds the
line as emitted, because the five formats agree on nothing and that is what the
per-application dashboards are for. And the level filter is partial by nature —
roughly 40% of lines are `unknown`, since access lines and Kong records carry
no severity for Loki to infer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One dashboard for all six gateways, with a selector accepting one, several or
all of them.
The selector has to bridge a mismatch: metrics carry the full gateway name
(catalog-dev) while audit logs carry the short domain (catalog), because the
former comes from the scrape job's label and the latter from the URI path each
gateway posts its http-log to. The variable holds the short form, PromQL
re-adds the -dev suffix and LogQL uses it as-is, so a single control drives
both signals. The capture group in the variable's regex also excludes the
unrelated `kong` gateway, which has no -dev suffix.
Latency is split three ways rather than shown as one number: total request
latency, time spent inside Kong, and upstream latency. The gap between the
first and the last is what the gateway itself costs, which is the question
worth asking of a proxy.
Status classes come from label_replace on the `code` label for metrics and
from LogQL's json parser for the audit records, since those carry no severity
of their own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploys the whole stack into the observability namespace of the local
cluster with Kustomize alone — no helm install, no dashboard created by
hand in the UI. The Makefile carries every command: deploy, dashboards,
diagnostics, PVC flush, teardown.
Dashboards are files, never UI objects. Each carries a stable uid, which
is what keeps URLs, cross-links and alerts alive across redeployments, and
references datasources by stable uid rather than by name. The dashboard
ConfigMap is generated without a name-suffix hash so applying it updates
it in place and Grafana reloads without a restart, while the provisioning
ConfigMaps keep their hash precisely so that a change there does force one.
Prometheus scrapes the kubelet /metrics/resource endpoint rather than
/metrics/cadvisor: on this cluster cadvisor exposes machine_* series only
and no container_* at all, which would leave every CPU and memory panel
empty.
Alloy ships every pod's logs to Loki through the Kubernetes API, so no
/var/log mount is needed. Tempo's metrics_generator feeds span metrics and
the service graph back into Prometheus, and the datasources are wired so a
log line's trace_id opens the matching trace.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland