Add a traces dashboard and make the span correlations work

Trace search lived only in Explore. `obs-traces` brings it into a dashboard:
TraceQL assembled from a service selector, a minimum duration, a status and a
free-text field, plus the aggregate RED view from the span metrics Tempo derives
from the traces themselves.

The status and free-text variables hold TraceQL fragments rather than values, so
an empty one contributes nothing and the query stays valid — the same reason the
log dashboards inject their search term the way they do.

Both span correlations were declared but non-functional.

tracesToLogsV2 mapped service.name onto Loki's `app` label, which Alloy derives
from app.kubernetes.io/name. That label exists for Keycloak and the Classic
Models API but not for the six Kong gateways or LibreChat — and the gateways are
the only place the trace id actually appears in the logs, so the mapping pointed
squarely at the services where it could not resolve. Replaced with a custom
query on ${__trace.traceId}, which matches any source writing the id anywhere in
the line.

tracesToMetrics had no queries at all and rendered a bare datasource link. It
now carries rate, error-rate and p95 against traces_spanmetrics_*, scoped to the
span's own service.

The service graph stays a table: serviceMap is a query type Grafana's frontend
assembles for Explore and returns no frames through the dashboard query API.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Stéphane Tailland
2026-08-26 07:44:57 +02:00
parent a2f5d725a3
commit 367ebd1da5
4 changed files with 1019 additions and 8 deletions
+45
View File
@@ -213,6 +213,51 @@ Each application dashboard carries the same four sections: an overview of RED
indicators, HTTP traffic, a runtime section specific to the stack in use, and a
**Logs** row, followed by container resources.
### Distributed tracing
`obs-traces` searches traces with TraceQL, assembled from a service selector, a
minimum duration, a status, and a free-text field for conditions the selectors
do not cover. The status and free-text variables hold TraceQL *fragments* rather
than values, so an empty one contributes nothing and the surrounding query stays
valid — "no filter" is the default rather than a special case.
Opening a trace gives the waterfall, every span's attributes and events, and
from a span its logs and its RED metrics. The rest of the dashboard is the
aggregate view: rate, errors and latency percentiles from the span metrics
Tempo's own `metrics_generator` derives from the traces, the slowest and most
failing operations by span name, and the service-to-service edges Tempo infers.
Span name rather than endpoint is the unit worth looking at: a slow database
statement surfaces next to a slow HTTP handler, which is what tracing buys over
plain HTTP metrics.
The interactive node graph stays in Explore — `serviceMap` is a query type
Grafana's frontend assembles and it returns nothing through the dashboard query
API, so the dashboard shows the same edges as a table instead.
#### Span-to-logs correlation
The Tempo datasource resolves a span's logs with a custom query on
`${__trace.traceId}` rather than by mapping `service.name` to a Loki label. The
label mapping it replaces resolved to `app`, which Alloy derives from
`app.kubernetes.io/name`: that exists for Keycloak and the Classic Models API
but not for the six Kong gateways — and the gateways were the only place the
trace id actually appeared. The mapping pointed at the services where it could
not work.
Coverage today, each verified by taking a trace id and searching for it:
| Service | Trace id in logs | How |
|---|---|---|
| Kong gateways (×6) | yes | `x-trace-id` in the http-log record |
| Keycloak | yes | MDC, via `KC_LOG_MDC_ENABLED` |
| classic-models-api | yes | inbound `traceparent` in the gunicorn access line |
| LibreChat | no | would need the upstream image patched |
`tracesToMetrics` likewise had no queries defined and rendered a bare datasource
link; it now carries rate, error-rate and p95 queries against the span metrics,
scoped to the span's own service.
### Log dashboards, split by kind
Logs are searched from three dashboards rather than one, because the three