A chat transaction is 212 spans, and almost all of them are Express
middleware, mongoose and mongodb calls, kong.dns lookups and raw SQL.
The flow — LibreChat to the gateway to the backend — is buried in it.
Set a default span filter on the panel: spanNameOperator "!~" against
those names with matchesOnly, so the view opens on the spans that
describe the exchange. "Show all spans" and clearing the filter both
bring them back.
Deliberately a display filter, not a source change. The spans are still
produced and still stored; only this panel's default view is quieter.
It is also the only lever that covers the LibreChat ones: LibreChat does
not use auto-instrumentations-node — it registers express, mongodb and
mongoose individually in its own code — so no environment variable
disables them, and they are the bulk of the 212.
The option names were checked against Grafana's own traces panel bundle
rather than assumed: spanName, spanNameOperator, serviceName,
serviceNameOperator, matchesOnly and criticalPathOnly all exist there,
and "!~" is among the operators it offers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The count-based filter let LibreChat's own /metrics scrapes through: 50
of them in six hours, crowding out the two real conversations. Combining
the count with a root-service match does not work either — TraceQL drops
the aggregate when it is joined to a spanset conjunction, verified: the
scrapes came back despite having a single server span.
Root on the service that starts a transaction instead, and require it to
reach a second one:
{trace:rootService="librechat"} && {resource.service.name!="librechat"}
Eight traces over six hours against fifty, and all eight are real: the
chat requests, the MCP OAuth callbacks, the OIDC login callback. Simpler
than counting spans and stricter.
The trade is explicit in the panel description: anything not entering
through LibreChat is invisible — a direct call to a gateway, a curl
against an API. "Everything" is still in the dropdown for those.
Worth revisiting when the agents replace LibreChat as the entry point,
since the root service is then no longer librechat.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kong does not propagate the trace when its openid-connect plugin calls
Keycloak: verified by tracing a request end to end and watching the
TOKEN_EXCHANGE it triggered land under Keycloak's own trace id instead.
It is not a missing setting — Kong's propagation module injects through
kong.service.request.*, which is the proxied upstream request, while the
plugin reaches Keycloak with its own HTTP client. Nothing in the
opentelemetry schema or kong_defaults.lua covers that path.
There is a join key even so: the audit record's
authenticated_entity.id is the same value as Keycloak's userId. So add a
panel listing the identities Kong authenticated on the selected trace —
usually more than one, LibreChat's own on the llm hop and the end user's
on the domain hops — and let a click on one scope the Keycloak panel to
that user.
A Grafana query variable cannot derive this automatically: Loki's
label/values endpoint rejects a pipeline ("only label matchers are
supported"), so the extracted field cannot feed a variable. Hence the
click.
That gives the intended chain for one user: LOGIN, CODE_TO_TOKEN,
USER_INFO_REQUEST, then a TOKEN_EXCHANGE per audience — verified against
live data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things were wrong with it, two of them mine.
Clicking a trace id emptied the trace list. The data link passed
var-traceFilter=${traceFilter:queryparam}, but that formatter already
emits "var-name=value" — prefixing it again wrote the literal string
"var-traceFilter={kind=server} | count() > 2" into the variable, which is
not valid TraceQL, so the panel errored. The link now sets only
var-traceId.
The spans were split across two tables, HTTP hops and Kong plugins. They
are one tree, and splitting them made the order unreadable. Replaced by
Grafana's own trace view: the full waterfall, every span nested in order
across LibreChat, Keycloak, the gateways and the backend, with span
attributes on click and the built-in jump to that span's logs.
And the dashboard never showed the identity side, which is most of the
point of following a request through this stack. Keycloak's event log
carries it: the token exchanges with the audience minted, the requesting
client, the end user, the grant type and the scopes. Parsed into columns
with one independent regexp per field — chaining json into line_format
into another parser has already cost a 60s timeout here, and each event
type carries a different subset of the keys, so an all-or-nothing parse
silently drops rows. The escaping is not incidental: the line is JSON, so
the message reads type=\"...\" and the pattern has to match the
backslash.
That panel is scoped to the time range rather than to the selected trace,
on purpose. Kong's openid-connect plugin calls Keycloak without
propagating the trace, so its token exchanges carry their own trace id
and scoping them to the transaction would show nothing. LibreChat's own
calls to Keycloak do propagate and do appear under the transaction —
verified both ways.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first version filtered the trace list by service name, which does not
remove the noise it was built to remove: Keycloak token calls and gateway
401s are services too, so with Service=All the panel showed exactly the
health-check traffic that makes Explore unusable, and the real LibreChat
transaction was nowhere in the first 50 rows.
Count SERVER spans instead. A token call has one, a health probe has one,
a gateway 401 has two; a real end-to-end transaction has four. So
{kind=server} | count() > 2 isolates them, and does it without naming a
single service — nothing to edit when an agent or a domain is added.
Verified against live data: it returns the LibreChat chat transactions
and nothing else.
The span panels were wrong in the same way: they showed the spans of
every trace at once, so the two tables were filled with unrelated
gateways. They now scope to {trace:id="$traceId"}, set by clicking a row
above, and sit empty until one is picked — which is the honest state, not
a wall of noise.
The service variable is gone; the trace filter is a named dropdown
instead, with the count-based default and a couple of looser options.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Explore's unfiltered trace search is unusable on this cluster: Tempo does
not return {} newest-first, so raising the limit does not walk backwards
in time and the health-check traffic — 91 Keycloak token calls and dozens
of /health/ and /metrics probes an hour — crowds real transactions out of
the first 20 results. Measured: at a fixed instant, limit=20 returned
06:50:43-06:51:07 while limit=100 returned 06:47:44-06:50:02.
Four panels:
* the traces on the dashboard's time range, filtered by service, each
row linking its trace id into the traceId variable;
* the HTTP hops between components — the SERVER-kind spans, one per
service entry, with method/status/route projected via select(), so
the Express middleware, DNS and SQL spans stay out;
* the Kong plugin spans with their individual durations;
* the Loki lines carrying the selected trace id.
No metric panels, on purpose. A span's own duration is already in the
span tables and is measured rather than estimated; aggregate RED by span
name is what obs-classic-models-api, obs-kong-gateways and obs-librechat
already show. Neither answers "the metrics of this span", so putting them
here only duplicated existing dashboards.
Two correlation limits are written into the panel descriptions rather
than left to be discovered: the join is by trace id and not span id
because no emitter logs a span id, and LibreChat logs carry no trace id
at all so its lines never appear.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland