Explore's unfiltered trace search is unusable on this cluster: Tempo does
not return {} newest-first, so raising the limit does not walk backwards
in time and the health-check traffic — 91 Keycloak token calls and dozens
of /health/ and /metrics probes an hour — crowds real transactions out of
the first 20 results. Measured: at a fixed instant, limit=20 returned
06:50:43-06:51:07 while limit=100 returned 06:47:44-06:50:02.
Four panels:
* the traces on the dashboard's time range, filtered by service, each
row linking its trace id into the traceId variable;
* the HTTP hops between components — the SERVER-kind spans, one per
service entry, with method/status/route projected via select(), so
the Express middleware, DNS and SQL spans stay out;
* the Kong plugin spans with their individual durations;
* the Loki lines carrying the selected trace id.
No metric panels, on purpose. A span's own duration is already in the
span tables and is measured rather than estimated; aggregate RED by span
name is what obs-classic-models-api, obs-kong-gateways and obs-librechat
already show. Neither answers "the metrics of this span", so putting them
here only duplicated existing dashboards.
Two correlation limits are written into the panel descriptions rather
than left to be discovered: the join is by trace id and not span id
because no emitter logs a span id, and LibreChat logs carry no trace id
at all so its lines never appear.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland
c95af8209e