Rebuild the trace dashboard around the trace itself

Three things were wrong with it, two of them mine.

Clicking a trace id emptied the trace list. The data link passed
var-traceFilter=${traceFilter:queryparam}, but that formatter already
emits "var-name=value" — prefixing it again wrote the literal string
"var-traceFilter={kind=server} | count() > 2" into the variable, which is
not valid TraceQL, so the panel errored. The link now sets only
var-traceId.

The spans were split across two tables, HTTP hops and Kong plugins. They
are one tree, and splitting them made the order unreadable. Replaced by
Grafana's own trace view: the full waterfall, every span nested in order
across LibreChat, Keycloak, the gateways and the backend, with span
attributes on click and the built-in jump to that span's logs.

And the dashboard never showed the identity side, which is most of the
point of following a request through this stack. Keycloak's event log
carries it: the token exchanges with the audience minted, the requesting
client, the end user, the grant type and the scopes. Parsed into columns
with one independent regexp per field — chaining json into line_format
into another parser has already cost a 60s timeout here, and each event
type carries a different subset of the keys, so an all-or-nothing parse
silently drops rows. The escaping is not incidental: the line is JSON, so
the message reads type=\"...\" and the pattern has to match the
backslash.

That panel is scoped to the time range rather than to the selected trace,
on purpose. Kong's openid-connect plugin calls Keycloak without
propagating the trace, so its token exchanges carry their own trace id
and scoping them to the transaction would show nothing. LibreChat's own
calls to Keycloak do propagate and do appear under the transaction —
verified both ways.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/grafana/dashboards/obs-distributed-traces.json

@@ -1,7 +1,7 @@
 {
   "uid": "obs-distributed-traces",
   "title": "Distributed Traces",
-  "description": "Find a real end-to-end transaction, then see only the spans worth looking at and the logs that go with it.\n\nBuilt because Explore's unfiltered search is unusable here: Tempo does not return {} newest-first, so raising the limit does not walk backwards in time, and health checks crowd real transactions out of the results.",
+  "description": "Follow one transaction end to end: pick it, read every span in order, see which tokens were issued, and read what was exchanged at each hop.",
   "tags": [
     "observability",
     "tracing",
@@ -21,25 +21,20 @@
     "list": [
       {
         "name": "traceFilter",
-        "label": "Trace filter",
+        "label": "Show",
         "type": "custom",
-        "query": "End-to-end — 3+ components : {kind=server} | count() > 2,Multi-component — 2+ : {kind=server} | count() > 1,Crossing a Kong gateway : {name=~\"kong\\\\..*\"} && {kind=server && resource.service.name!~\".*-dev\"},Everything (noisy) : {}",
+        "query": "Transactions — 3+ services : {kind=server} | count() > 2,2+ services : {kind=server} | count() > 1,Everything (noisy) : {}",
         "options": [
           {
-            "text": "End-to-end — 3+ components",
+            "text": "Transactions — 3+ services",
             "value": "{kind=server} | count() > 2",
             "selected": true
           },
           {
-            "text": "Multi-component — 2+",
+            "text": "2+ services",
             "value": "{kind=server} | count() > 1",
             "selected": false
           },
-          {
-            "text": "Crossing a Kong gateway",
-            "value": "{name=~\"kong\\\\..*\"} && {kind=server && resource.service.name!~\".*-dev\"}",
-            "selected": false
-          },
           {
             "text": "Everything (noisy)",
             "value": "{}",
@@ -47,12 +42,12 @@
           }
         ],
         "current": {
-          "text": "End-to-end — 3+ components",
+          "text": "Transactions — 3+ services",
           "value": "{kind=server} | count() > 2"
         },
         "includeAll": false,
         "multi": false,
-        "description": "Which traces count as interesting. Counting SERVER spans is what separates real transactions from token calls and health probes."
+        "description": "Which traces are worth listing."
       },
       {
         "name": "traceId",
@@ -63,14 +58,14 @@
           "text": "",
           "value": ""
         },
-        "description": "Set by clicking a Trace ID above, or paste one. Empty leaves the span panels empty."
+        "description": "Set by clicking a Trace ID above, or paste one."
       }
     ]
   },
   "panels": [
     {
       "type": "row",
-      "title": "Traces",
+      "title": "1 — Pick a transaction",
       "gridPos": {
         "h": 1,
         "w": 24,
@@ -82,14 +77,14 @@
     },
     {
       "type": "table",
-      "title": "End-to-end transactions",
+      "title": "Transactions",
-      "description": "Default filter is {kind=server} | count() > 2 — traces where at least three components received an HTTP request. That is what an end-to-end transaction is here, and it needs no service name hardcoded.\n\nIt exists because filtering by service does NOT remove the noise: Keycloak token calls and gateway 401s are services too. Counting server spans does — a token call has one, a health probe has one, a real LibreChat transaction has four.\n\nClick a Trace ID: the panels below scope to it.",
+      "description": "Traces where at least three services received an HTTP request — a real end-to-end flow rather than a token call or a health probe. Filtering by service name does not achieve this: Keycloak and the gateways are services too.\n\nClick a Trace ID to load it below.",
       "datasource": {
         "type": "tempo",
         "uid": "tempo"
       },
       "gridPos": {
-        "h": 10,
+        "h": 8,
         "w": 24,
         "x": 0,
         "y": 1
@@ -104,8 +99,7 @@
           "queryType": "traceql",
           "query": "$traceFilter",
           "tableType": "traces",
-          "limit": 50,
+          "limit": 50
-          "spss": 50
         }
       ],
       "fieldConfig": {
@@ -126,8 +120,8 @@
                 "id": "links",
                 "value": [
                   {
-                    "title": "Scope the panels below to this trace",
+                    "title": "Open this trace below",
-                    "url": "d/obs-distributed-traces/distributed-traces?${__url_time_range}&var-traceId=${__value.raw}&var-traceFilter=${traceFilter:queryparam}"
+                    "url": "d/obs-distributed-traces/distributed-traces?${__url_time_range}&var-traceId=${__value.raw}"
                   }
                 ]
               }
@@ -147,42 +141,117 @@
     },
     {
       "type": "row",
-      "title": "Spans of the selected trace",
+      "title": "2 — Every span of the trace, in order",
       "gridPos": {
         "h": 1,
         "w": 24,
         "x": 0,
-        "y": 11
+        "y": 9
+      },
+      "collapsed": false,
+      "panels": []
+    },
+    {
+      "type": "traces",
+      "title": "Trace $traceId",
+      "description": "Grafana's own trace view: the full waterfall, every span nested in order, across LibreChat, Keycloak, the Kong gateways and the backend — no split between HTTP hops and plugins, they are one tree.\n\nClick a span to expand its attributes: method, route, status, durations, and whatever the instrumentation attached. From a span, 'Logs for this span' jumps to Loki.",
+      "datasource": {
+        "type": "tempo",
+        "uid": "tempo"
+      },
+      "gridPos": {
+        "h": 22,
+        "w": 24,
+        "x": 0,
+        "y": 10
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": {
+            "type": "tempo",
+            "uid": "tempo"
+          },
+          "queryType": "traceql",
+          "query": "$traceId",
+          "limit": 300
+        }
+      ]
+    },
+    {
+      "type": "row",
+      "title": "3 — Identity: what was issued, to whom, for what",
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 32
       },
       "collapsed": false,
       "panels": []
     },
     {
       "type": "table",
-      "title": "HTTP hops between components",
+      "title": "Keycloak identity events",
-      "description": "The SERVER-kind spans of the selected trace: one row per component that received the request. Browser to LibreChat, LibreChat to the llm gateway, gateway to a domain gateway, gateway to the backend. Express middleware, DNS and SQL spans are excluded on purpose.\n\nEmpty until a Trace ID is selected above.",
+      "description": "Parsed from Keycloak's own event log: the token exchanges, logins and code-to-token swaps, with the audience minted, the requesting client, the end user and the grant type. This is where the OIDC substance lives — the Authorization header itself is REDACTED in the Kong audit records, so claims cannot be read from there.\n\nScoped to the dashboard TIME RANGE, not to the selected trace, and deliberately: Kong's openid-connect plugin calls Keycloak without propagating the trace, so those token exchanges carry their own trace id. Scoping them to the transaction would show nothing. LibreChat's own calls to Keycloak DO propagate and do appear under the transaction's trace.",
       "datasource": {
-        "type": "tempo",
+        "type": "loki",
-        "uid": "tempo"
+        "uid": "loki"
       },
       "gridPos": {
-        "h": 10,
+        "h": 11,
-        "w": 12,
+        "w": 24,
         "x": 0,
-        "y": 12
+        "y": 33
       },
       "targets": [
         {
           "refId": "A",
           "datasource": {
-            "type": "tempo",
+            "type": "loki",
-            "uid": "tempo"
+            "uid": "loki"
           },
-          "queryType": "traceql",
+          "editorMode": "code",
-          "query": "{trace:id=\"$traceId\" && kind=server} | select(span.http.method, span.http.status_code, span.http.route, span.http.url)",
+          "queryType": "range",
-          "tableType": "spans",
+          "expr": "{job=\"keycloak/keycloak\"} |= \"org.keycloak.events\" | regexp `type=\\\\\"(?P<event>[^\\\\\"]*)\\\\\"` | regexp `clientId=\\\\\"(?P<client>[^\\\\\"]*)\\\\\"` | regexp `audience=\\\\\"(?P<audience>[^\\\\\"]*)\\\\\"` | regexp `username=\\\\\"(?P<user>[^\\\\\"]*)\\\\\"` | regexp `grant_type=\\\\\"(?P<grant>[^\\\\\"]*)\\\\\"` | regexp `scope=\\\\\"(?P<scope>[^\\\\\"]*)\\\\\"`"
-          "limit": 50,
+        }
-          "spss": 50
+      ],
+      "transformations": [
+        {
+          "id": "extractFields",
+          "options": {
+            "source": "labels",
+            "replace": false
+          }
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "labels": true,
+              "Line": true,
+              "tsNs": true,
+              "id": true,
+              "Time": false
+            },
+            "indexByName": {
+              "Time": 0,
+              "event": 1,
+              "client": 2,
+              "audience": 3,
+              "user": 4,
+              "grant": 5,
+              "scope": 6
+            },
+            "renameByName": {
+              "event": "Event",
+              "client": "Client",
+              "audience": "Audience",
+              "user": "User",
+              "grant": "Grant type",
+              "scope": "Scopes"
+            }
+          }
         }
       ],
       "fieldConfig": {
@@ -195,75 +264,40 @@
         "overrides": []
       },
       "options": {
-        "showHeader": true
+        "showHeader": true,
-      }
+        "sortBy": [
-    },
+          {
-    {
+            "displayName": "Time",
-      "type": "table",
+            "desc": true
-      "title": "Kong plugin execution",
-      "description": "Every plugin Kong ran on the selected trace, with its own duration — openid-connect, rate-limiting, the ai-* plugins, opentelemetry. This is what shows which plugin costs latency on a gateway hop.\n\nEmpty until a Trace ID is selected above.",
-      "datasource": {
-        "type": "tempo",
-        "uid": "tempo"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 12,
-        "x": 12,
-        "y": 12
-      },
-      "targets": [
-        {
-          "refId": "A",
-          "datasource": {
-            "type": "tempo",
-            "uid": "tempo"
-          },
-          "queryType": "traceql",
-          "query": "{trace:id=\"$traceId\" && name=~\"kong\\\\..*plugin.*\"}",
-          "tableType": "spans",
-          "limit": 50,
-          "spss": 50
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "align": "auto",
-            "filterable": true
           }
-        },
+        ]
-        "overrides": []
-      },
-      "options": {
-        "showHeader": true
       }
     },
     {
       "type": "row",
-      "title": "Logs of the selected trace",
+      "title": "4 — What was actually exchanged",
       "gridPos": {
         "h": 1,
         "w": 24,
         "x": 0,
-        "y": 22
+        "y": 44
       },
       "collapsed": false,
       "panels": []
     },
     {
       "type": "logs",
-      "title": "Logs carrying trace $traceId",
+      "title": "Logs of trace $traceId",
-      "description": "Correlation is by TRACE id, not span id: no emitter logs a span id, so this is every line of the whole trace rather than of one span.\n\nCoverage is uneven: kong-audit and the backend's access log carry the id on every line, and the backend's application log does too since v1.0.4. LibreChat logs carry no trace id at all, so its lines never appear here.\n\nWith traceId empty the filter matches everything — pick a trace above.",
+      "description": "Every log line carrying this trace id. The Kong audit records are the substance: request headers, response status, per-stage latencies, and on the domain gateways request_body and response_body — so the MCP tools/call payload and its result are readable in full. Expand a line to read it as fields.\n\nOne gap, and it is a capture gap rather than a display one: the llm gateway has no body capture, so LLM prompts and completions are not recorded.",
       "datasource": {
         "type": "loki",
         "uid": "loki"
       },
       "gridPos": {
-        "h": 12,
+        "h": 14,
         "w": 24,
         "x": 0,
-        "y": 23
+        "y": 45
       },
       "targets": [
         {
@@ -279,7 +313,8 @@
         "showTime": true,
         "wrapLogMessage": true,
         "sortOrder": "Ascending",
-        "enableLogDetails": true
+        "enableLogDetails": true,
+        "prettifyLogMessage": true
       }
     }
   ]