Select transactions by what starts them, not by span count

The count-based filter let LibreChat's own /metrics scrapes through: 50
of them in six hours, crowding out the two real conversations. Combining
the count with a root-service match does not work either — TraceQL drops
the aggregate when it is joined to a spanset conjunction, verified: the
scrapes came back despite having a single server span.

Root on the service that starts a transaction instead, and require it to
reach a second one:

  {trace:rootService="librechat"} && {resource.service.name!="librechat"}

Eight traces over six hours against fifty, and all eight are real: the
chat requests, the MCP OAuth callbacks, the OIDC login callback. Simpler
than counting spans and stricter.

The trade is explicit in the panel description: anything not entering
through LibreChat is invisible — a direct call to a gateway, a curl
against an API. "Everything" is still in the dropdown for those.

Worth revisiting when the agents replace LibreChat as the entry point,
since the root service is then no longer librechat.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/grafana/dashboards/obs-distributed-traces.json

@@ -23,31 +23,36 @@
         "name": "traceFilter",
         "label": "Show",
         "type": "custom",
-        "query": "Transactions — 3+ services : {kind=server} | count() > 2,2+ services : {kind=server} | count() > 1,Everything (noisy) : {}",
+        "query": "LibreChat transactions : {trace:rootService=\"librechat\"} && {resource.service.name!=\"librechat\"},Reaching a Kong gateway : {trace:rootService=\"librechat\"} && {name=~\"kong\\\\..*\"},All LibreChat traces (incl. /metrics scrapes) : {trace:rootService=\"librechat\"},Everything (very noisy) : {}",
         "options": [
           {
-            "text": "Transactions — 3+ services",
+            "text": "LibreChat transactions",
-            "value": "{kind=server} | count() > 2",
+            "value": "{trace:rootService=\"librechat\"} && {resource.service.name!=\"librechat\"}",
             "selected": true
           },
           {
-            "text": "2+ services",
+            "text": "Reaching a Kong gateway",
-            "value": "{kind=server} | count() > 1",
+            "value": "{trace:rootService=\"librechat\"} && {name=~\"kong\\\\..*\"}",
             "selected": false
           },
           {
-            "text": "Everything (noisy)",
+            "text": "All LibreChat traces (incl. /metrics scrapes)",
+            "value": "{trace:rootService=\"librechat\"}",
+            "selected": false
+          },
+          {
+            "text": "Everything (very noisy)",
             "value": "{}",
             "selected": false
           }
         ],
         "current": {
-          "text": "Transactions — 3+ services",
+          "text": "LibreChat transactions",
-          "value": "{kind=server} | count() > 2"
+          "value": "{trace:rootService=\"librechat\"} && {resource.service.name!=\"librechat\"}"
         },
         "includeAll": false,
         "multi": false,
-        "description": "Which traces are worth listing."
+        "description": "Traces started BY LibreChat and leaving it. Rooting on LibreChat is what makes this a user-facing transaction rather than infrastructure chatter; requiring a second service is what drops Prometheus scraping LibreChat's own /metrics, which otherwise fills the list."
       },
       {
         "name": "traceId",
@@ -89,7 +94,7 @@
     {
       "type": "table",
       "title": "Transactions",
-      "description": "Traces where at least three services received an HTTP request — a real end-to-end flow rather than a token call or a health probe. Filtering by service name does not achieve this: Keycloak and the gateways are services too.\n\nClick a Trace ID to load it below.",
+      "description": "Traces LibreChat started that reached another service — the user-facing transactions.\n\nCounting spans was the previous approach and it let LibreChat's own /metrics scrapes through: with 50 of them in six hours they crowded out the two real conversations. Rooting on the service that starts a transaction is both simpler and stricter.\n\nThe trade: anything not entering through LibreChat is invisible here — a direct call to a gateway, or a curl against an API. Use 'Everything' for those.\n\nClick a Trace ID to load it below.",
       "datasource": {
         "type": "tempo",
         "uid": "tempo"