Three things were wrong with it, two of them mine.
Clicking a trace id emptied the trace list. The data link passed
var-traceFilter=${traceFilter:queryparam}, but that formatter already
emits "var-name=value" — prefixing it again wrote the literal string
"var-traceFilter={kind=server} | count() > 2" into the variable, which is
not valid TraceQL, so the panel errored. The link now sets only
var-traceId.
The spans were split across two tables, HTTP hops and Kong plugins. They
are one tree, and splitting them made the order unreadable. Replaced by
Grafana's own trace view: the full waterfall, every span nested in order
across LibreChat, Keycloak, the gateways and the backend, with span
attributes on click and the built-in jump to that span's logs.
And the dashboard never showed the identity side, which is most of the
point of following a request through this stack. Keycloak's event log
carries it: the token exchanges with the audience minted, the requesting
client, the end user, the grant type and the scopes. Parsed into columns
with one independent regexp per field — chaining json into line_format
into another parser has already cost a 60s timeout here, and each event
type carries a different subset of the keys, so an all-or-nothing parse
silently drops rows. The escaping is not incidental: the line is JSON, so
the message reads type=\"...\" and the pattern has to match the
backslash.
That panel is scoped to the time range rather than to the selected trace,
on purpose. Kong's openid-connect plugin calls Keycloak without
propagating the trace, so its token exchanges carry their own trace id
and scoping them to the transaction would show nothing. LibreChat's own
calls to Keycloak do propagate and do appear under the transaction —
verified both ways.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first version filtered the trace list by service name, which does not
remove the noise it was built to remove: Keycloak token calls and gateway
401s are services too, so with Service=All the panel showed exactly the
health-check traffic that makes Explore unusable, and the real LibreChat
transaction was nowhere in the first 50 rows.
Count SERVER spans instead. A token call has one, a health probe has one,
a gateway 401 has two; a real end-to-end transaction has four. So
{kind=server} | count() > 2 isolates them, and does it without naming a
single service — nothing to edit when an agent or a domain is added.
Verified against live data: it returns the LibreChat chat transactions
and nothing else.
The span panels were wrong in the same way: they showed the spans of
every trace at once, so the two tables were filled with unrelated
gateways. They now scope to {trace:id="$traceId"}, set by clicking a row
above, and sit empty until one is picked — which is the honest state, not
a wall of noise.
The service variable is gone; the trace filter is a named dropdown
instead, with the count-based default and a couple of looser options.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Explore's unfiltered trace search is unusable on this cluster: Tempo does
not return {} newest-first, so raising the limit does not walk backwards
in time and the health-check traffic — 91 Keycloak token calls and dozens
of /health/ and /metrics probes an hour — crowds real transactions out of
the first 20 results. Measured: at a fixed instant, limit=20 returned
06:50:43-06:51:07 while limit=100 returned 06:47:44-06:50:02.
Four panels:
* the traces on the dashboard's time range, filtered by service, each
row linking its trace id into the traceId variable;
* the HTTP hops between components — the SERVER-kind spans, one per
service entry, with method/status/route projected via select(), so
the Express middleware, DNS and SQL spans stay out;
* the Kong plugin spans with their individual durations;
* the Loki lines carrying the selected trace id.
No metric panels, on purpose. A span's own duration is already in the
span tables and is measured rather than estimated; aggregate RED by span
name is what obs-classic-models-api, obs-kong-gateways and obs-librechat
already show. Neither answers "the metrics of this span", so putting them
here only duplicated existing dashboards.
Two correlation limits are written into the panel descriptions rather
than left to be discovered: the join is by trace id and not span id
because no emitter logs a span id, and LibreChat logs carry no trace id
at all so its lines never appear.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland