Exposes Prometheus, Loki and Tempo as tools an LLM can call, so a question
like "what failed in the last transaction" is answered by querying rather
than by building another panel — which is where the trace dashboard kept
running out of road.
Works against OSS Grafana; the Cloud-only tools (Incidents, OnCall, Sift,
Agent Observability, Assistant) are simply absent. Verified end to end:
the MCP handshake succeeds and list_datasources returns Loki, Prometheus
and Tempo.
Authenticates with a Grafana service account token, Viewer role, created
once through /api/serviceaccounts and stored in the gitignored
k8s/secret.yaml. secret.example.yaml documents how to mint it, since
Grafana returns the value only at creation.
Two things the deployment needs that are not obvious:
* runAsUser must be numeric. The image declares USER mcp-grafana by
name, Kubernetes cannot resolve that to check it is not root, and
refuses to start the container under runAsNonRoot with "image has
non-numeric user". 1000/999 is what the image actually uses.
* --allowed-hosts must name the Service. The server validates the Host
header against a loopback-only default as DNS-rebinding protection, so
the first in-cluster call returned "forbidden: host not allowed".
Named rather than disabled with "*": the check is worth keeping.
Deliberately in-cluster only, with no OIDC and no audit trail — anything
that can reach this Service reads all the telemetry. The intended end
state is the same treatment as the five domain MCP servers, behind a Kong
gateway with openid-connect, consumer groups and http-log.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
I set spanNameOperator "!~" with a pattern and said it was verified. It
was not: I had checked that the option names exist in Grafana's bundle,
not that the behaviour was what I claimed, and then wrote that it was no
longer a supposition. It shipped a filter chip that hid nothing — the
panel reported "212 matches" with every middleware span still listed.
Reading the implementation: spanName is compared with an exact string
equality against operationName, so only "=" and "!=" against a single
literal name do anything. It is also pushed into the tag-filter list,
where "=~"/"!~" are honoured — but that matcher looks the key up among
the span's tags, and a span's name is not a tag, so the pattern matches
nothing and the negative filter passes everything.
So pattern-based exclusion is not available in this panel. Remove the
option rather than leave a chip that suggests filtering is happening, and
say so in the description, pointing at the built-in Critical path button
as the one lever that does narrow the view.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chat transaction is 212 spans, and almost all of them are Express
middleware, mongoose and mongodb calls, kong.dns lookups and raw SQL.
The flow — LibreChat to the gateway to the backend — is buried in it.
Set a default span filter on the panel: spanNameOperator "!~" against
those names with matchesOnly, so the view opens on the spans that
describe the exchange. "Show all spans" and clearing the filter both
bring them back.
Deliberately a display filter, not a source change. The spans are still
produced and still stored; only this panel's default view is quieter.
It is also the only lever that covers the LibreChat ones: LibreChat does
not use auto-instrumentations-node — it registers express, mongodb and
mongoose individually in its own code — so no environment variable
disables them, and they are the bulk of the 212.
The option names were checked against Grafana's own traces panel bundle
rather than assumed: spanName, spanNameOperator, serviceName,
serviceNameOperator, matchesOnly and criticalPathOnly all exist there,
and "!~" is among the operators it offers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The count-based filter let LibreChat's own /metrics scrapes through: 50
of them in six hours, crowding out the two real conversations. Combining
the count with a root-service match does not work either — TraceQL drops
the aggregate when it is joined to a spanset conjunction, verified: the
scrapes came back despite having a single server span.
Root on the service that starts a transaction instead, and require it to
reach a second one:
{trace:rootService="librechat"} && {resource.service.name!="librechat"}
Eight traces over six hours against fifty, and all eight are real: the
chat requests, the MCP OAuth callbacks, the OIDC login callback. Simpler
than counting spans and stricter.
The trade is explicit in the panel description: anything not entering
through LibreChat is invisible — a direct call to a gateway, a curl
against an API. "Everything" is still in the dropdown for those.
Worth revisiting when the agents replace LibreChat as the entry point,
since the root service is then no longer librechat.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kong does not propagate the trace when its openid-connect plugin calls
Keycloak: verified by tracing a request end to end and watching the
TOKEN_EXCHANGE it triggered land under Keycloak's own trace id instead.
It is not a missing setting — Kong's propagation module injects through
kong.service.request.*, which is the proxied upstream request, while the
plugin reaches Keycloak with its own HTTP client. Nothing in the
opentelemetry schema or kong_defaults.lua covers that path.
There is a join key even so: the audit record's
authenticated_entity.id is the same value as Keycloak's userId. So add a
panel listing the identities Kong authenticated on the selected trace —
usually more than one, LibreChat's own on the llm hop and the end user's
on the domain hops — and let a click on one scope the Keycloak panel to
that user.
A Grafana query variable cannot derive this automatically: Loki's
label/values endpoint rejects a pipeline ("only label matchers are
supported"), so the extracted field cannot feed a variable. Hence the
click.
That gives the intended chain for one user: LOGIN, CODE_TO_TOKEN,
USER_INFO_REQUEST, then a TOKEN_EXCHANGE per audience — verified against
live data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things were wrong with it, two of them mine.
Clicking a trace id emptied the trace list. The data link passed
var-traceFilter=${traceFilter:queryparam}, but that formatter already
emits "var-name=value" — prefixing it again wrote the literal string
"var-traceFilter={kind=server} | count() > 2" into the variable, which is
not valid TraceQL, so the panel errored. The link now sets only
var-traceId.
The spans were split across two tables, HTTP hops and Kong plugins. They
are one tree, and splitting them made the order unreadable. Replaced by
Grafana's own trace view: the full waterfall, every span nested in order
across LibreChat, Keycloak, the gateways and the backend, with span
attributes on click and the built-in jump to that span's logs.
And the dashboard never showed the identity side, which is most of the
point of following a request through this stack. Keycloak's event log
carries it: the token exchanges with the audience minted, the requesting
client, the end user, the grant type and the scopes. Parsed into columns
with one independent regexp per field — chaining json into line_format
into another parser has already cost a 60s timeout here, and each event
type carries a different subset of the keys, so an all-or-nothing parse
silently drops rows. The escaping is not incidental: the line is JSON, so
the message reads type=\"...\" and the pattern has to match the
backslash.
That panel is scoped to the time range rather than to the selected trace,
on purpose. Kong's openid-connect plugin calls Keycloak without
propagating the trace, so its token exchanges carry their own trace id
and scoping them to the transaction would show nothing. LibreChat's own
calls to Keycloak do propagate and do appear under the transaction —
verified both ways.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first version filtered the trace list by service name, which does not
remove the noise it was built to remove: Keycloak token calls and gateway
401s are services too, so with Service=All the panel showed exactly the
health-check traffic that makes Explore unusable, and the real LibreChat
transaction was nowhere in the first 50 rows.
Count SERVER spans instead. A token call has one, a health probe has one,
a gateway 401 has two; a real end-to-end transaction has four. So
{kind=server} | count() > 2 isolates them, and does it without naming a
single service — nothing to edit when an agent or a domain is added.
Verified against live data: it returns the LibreChat chat transactions
and nothing else.
The span panels were wrong in the same way: they showed the spans of
every trace at once, so the two tables were filled with unrelated
gateways. They now scope to {trace:id="$traceId"}, set by clicking a row
above, and sit empty until one is picked — which is the honest state, not
a wall of noise.
The service variable is gone; the trace filter is a named dropdown
instead, with the count-based default and a couple of looser options.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Explore's unfiltered trace search is unusable on this cluster: Tempo does
not return {} newest-first, so raising the limit does not walk backwards
in time and the health-check traffic — 91 Keycloak token calls and dozens
of /health/ and /metrics probes an hour — crowds real transactions out of
the first 20 results. Measured: at a fixed instant, limit=20 returned
06:50:43-06:51:07 while limit=100 returned 06:47:44-06:50:02.
Four panels:
* the traces on the dashboard's time range, filtered by service, each
row linking its trace id into the traceId variable;
* the HTTP hops between components — the SERVER-kind spans, one per
service entry, with method/status/route projected via select(), so
the Express middleware, DNS and SQL spans stay out;
* the Kong plugin spans with their individual durations;
* the Loki lines carrying the selected trace id.
No metric panels, on purpose. A span's own duration is already in the
span tables and is measured rather than estimated; aggregate RED by span
name is what obs-classic-models-api, obs-kong-gateways and obs-librechat
already show. Neither answers "the metrics of this span", so putting them
here only duplicated existing dashboards.
Two correlation limits are written into the panel descriptions rather
than left to be discovered: the join is by trace id and not span id
because no emitter logs a span id, and LibreChat logs carry no trace id
at all so its lines never appear.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dashboard reimplemented, less well, what Explore already does: its search
panel had no TraceQL query builder and no autocompletion, and its service
dependency table replaced an interactive node graph with a list. Searching and
exploring traces belongs in Explore.
The datasource correlations this work also fixed are kept — they are what was
actually missing, and they apply in Explore exactly as they would in a
dashboard.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Trace search lived only in Explore. `obs-traces` brings it into a dashboard:
TraceQL assembled from a service selector, a minimum duration, a status and a
free-text field, plus the aggregate RED view from the span metrics Tempo derives
from the traces themselves.
The status and free-text variables hold TraceQL fragments rather than values, so
an empty one contributes nothing and the query stays valid — the same reason the
log dashboards inject their search term the way they do.
Both span correlations were declared but non-functional.
tracesToLogsV2 mapped service.name onto Loki's `app` label, which Alloy derives
from app.kubernetes.io/name. That label exists for Keycloak and the Classic
Models API but not for the six Kong gateways or LibreChat — and the gateways are
the only place the trace id actually appears in the logs, so the mapping pointed
squarely at the services where it could not resolve. Replaced with a custom
query on ${__trace.traceId}, which matches any source writing the id anywhere in
the line.
tracesToMetrics had no queries at all and rendered a bare datasource link. It
now carries rate, error-rate and p95 against traces_spanmetrics_*, scoped to the
span's own service.
The service graph stays a table: serviceMap is a query type Grafana's frontend
assembles for Explore and returns no frames through the dashboard query API.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The panel never displayed: its query combined `line_format` with `logfmt`, and
on this data that pairing is pathological. Measured on a 24h range, the stream
selector with its json filter answers in 0.7s, adding line_format alone 0.3s,
adding logfmt alone 0.2s — chained, 60s and a timeout. The bar panels above it
survived because an aggregation streams; the log query did not.
Replaced with one independent `regexp` per field, reading the key="value" pairs
straight out of the JSON message without reformatting the line. 0.1s, and the
raw record now stays raw since nothing rewrites it.
Independent stages also fix a correctness problem the single-regexp version
would have had: the field set differs per event type — LOGIN_ERROR carries
`error` and `grant_type`, USER_SESSION_DELETED carries neither — so one regexp
spanning them all would drop whole rows on a missing field. All four types
observed on this cluster now extract correctly.
Failed and successful logins widen from LOGIN_ERROR and LOGIN to the *_ERROR
and CODE_TO_TOKEN families: a failed refresh or code exchange is the same
signal, and TOKEN_EXCHANGE dominates this cluster's traffic.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three defects, all mine.
Column order was ignored because organize's indexByName was keyed on the
renamed display labels. Grafana keys it on the original field names, so the
mapping matched nothing and the order fell back to whatever order the frames
arrived in — which is why the raw line landed second. Every table is affected,
not just the access one.
The byte count was wrong for the NCSA sources: a trailing `pattern` capture
takes the rest of the line, so `<size>` held `16 "-" "kube-probe/1.35"` rather
than 16, and for nginx it swallowed the whole referer and user-agent. Replaced
with a `regexp` parser bounding the capture on \d+, which also handles both
line shapes — Keycloak's access line ends at the size while gunicorn's and
nginx's carry more fields after it.
Access log columns are now time, status, method, path, source, client, latency,
bytes, raw line, with the raw record last in every table.
gunicorn and nginx now share one query since their format is identical, which
also halves the duplicate class names the status panel showed — one series per
query returning the same class.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One consolidated search compromised on all three questions at once. It also
had a real defect: the Source selector carried `.+` as its All value, so
choosing "All" widened the query back to every stream in the cluster and the
scope preset only ever populated the picker. MongoDB and cert-manager showed up
in what was meant to be an application view.
Each query now carries both `job=~"${source:regex}"` and a literal job matcher,
so the selector actually restricts rather than widens.
The three kinds are subtracted from each other rather than merely filtered:
application logs exclude Keycloak's access-log and events loggers by name, the
Django container's NCSA lines by shape, and the Kong containers' stray access
lines by requiring the nginx date prefix.
Levels are read from the line rather than taken from Loki's detected_level.
That inference marks 78% of the Kong gateway output `unknown` even though every
one of those lines states [notice] or [warn], so filtering on it hid most of
the gateway's own logs. Each parser extracts the level and lowercases it, which
puts nginx's notice, Keycloak's INFO and LibreChat's info on one scale — and
removes the need for an `unknown` entry in the selector.
Applies move to server-side. Client-side apply stores a full copy of each
object in the last-applied-configuration annotation, capped at 256KB, and the
dashboard ConfigMap passed that at eight dashboards; the failure surfaced as
"metadata.annotations: Too long". Server-side apply tracks field ownership in
metadata instead, moving the ceiling to etcd's ~1MB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The per-application dashboards answer "how is this component behaving". This
one answers "where did that string come from", across every source at once, and
is kept separate so neither has to compromise for the other.
Scope is a curated preset rather than a free namespace picker: the cluster
carries 48 log sources and most are infrastructure noise. The presets are job
regexes over the <namespace>/<container> label Alloy sets — except kong-audit,
which Fluent Bit ships with no namespace at all. That mismatch between the two
collection paths is the reason a single naive selector cannot span both, and
why the presets spell the sources out.
The search term is injected inside LogQL backticks, a raw string, so a quote
typed in the box cannot break the query; an empty box yields `(?i)`, a valid
regex matching every line, which makes "no search" the default rather than a
special case. The four stat panels carry `or vector(0)` so an empty result
reads as 0 rather than "No data" — on a search box, finding nothing is an
answer, not a failure.
Two deliberate omissions. No per-format parsing: the Message column holds the
line as emitted, because the five formats agree on nothing and that is what the
per-application dashboards are for. And the level filter is partial by nature —
roughly 40% of lines are `unknown`, since access lines and Kong records carry
no severity for Loki to infer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Raw lines are close to unreadable when the payload is JSON, and the Kong audit
records are the worst case: a wall of braces where the interesting fields are
status, route and latency.
Each dashboard now leads with a table carrying one column per parsed field,
sortable and filterable, with the unparsed record as the last column and the
cell inspector enabled on every cell. The original logs panel moves into a
collapsed row underneath rather than disappearing: a table cannot render ANSI
colour, expand a record inline, tail live, or turn a trace id into a link to
Tempo.
Parsing is per-format since the five sources agree on nothing — nested JSON for
Kong, JSON for Keycloak, NCSA access lines for the Django API, ANSI-coloured
text for LibreChat, logfmt for the stack itself.
The Kong and stack parsers name their fields explicitly rather than extracting
everything: a bare `| json` on a Kong record yields around seventy columns, and
a bare `| logfmt` explodes Loki's own query-metrics lines into dozens. Columns
are selected by include-pattern rather than by excluding names, because the
stream labels that ride along are not a fixed set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One dashboard for all six gateways, with a selector accepting one, several or
all of them.
The selector has to bridge a mismatch: metrics carry the full gateway name
(catalog-dev) while audit logs carry the short domain (catalog), because the
former comes from the scrape job's label and the latter from the URI path each
gateway posts its http-log to. The variable holds the short form, PromQL
re-adds the -dev suffix and LogQL uses it as-is, so a single control drives
both signals. The capture group in the variable's regex also excludes the
unrelated `kong` gateway, which has no -dev suffix.
Latency is split three ways rather than shown as one number: total request
latency, time spent inside Kong, and upstream latency. The gap between the
first and the last is what the gateway itself costs, which is the question
worth asking of a proxy.
Status classes come from label_replace on the `code` label for metrics and
from LogQL's json parser for the audit records, since those carry no severity
of their own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every dataplane pod the Kong operator creates declares a container port named
`metrics` (8100) and carries the gateway-name label, so discovery keys off
those rather than an annotation — the operator owns that pod template, not us,
and adding annotations there would mean editing the shared dataplane template
and restarting every gateway.
Keeping on the port NAME rather than a number also drops the init containers,
which declare no port at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploys the whole stack into the observability namespace of the local
cluster with Kustomize alone — no helm install, no dashboard created by
hand in the UI. The Makefile carries every command: deploy, dashboards,
diagnostics, PVC flush, teardown.
Dashboards are files, never UI objects. Each carries a stable uid, which
is what keeps URLs, cross-links and alerts alive across redeployments, and
references datasources by stable uid rather than by name. The dashboard
ConfigMap is generated without a name-suffix hash so applying it updates
it in place and Grafana reloads without a restart, while the provisioning
ConfigMaps keep their hash precisely so that a change there does force one.
Prometheus scrapes the kubelet /metrics/resource endpoint rather than
/metrics/cadvisor: on this cluster cadvisor exposes machine_* series only
and no container_* at all, which would leave every CPU and memory panel
empty.
Alloy ships every pod's logs to Loki through the Kubernetes API, so no
/var/log mount is needed. Tempo's metrics_generator feeds span metrics and
the service graph back into Prometheus, and the datasources are wired so a
log line's trace_id opens the matching trace.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland