From e6a3c53c8b95379adca9a69851c9096811ac0854 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?St=C3=A9phane=20Tailland?= <stephane@sttlab.eu>
Date: Thu, 27 Aug 2026 13:45:19 +0200
Subject: [PATCH] Drop the span filter that could not work
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

I set spanNameOperator "!~" with a pattern and said it was verified. It
was not: I had checked that the option names exist in Grafana's bundle,
not that the behaviour was what I claimed, and then wrote that it was no
longer a supposition. It shipped a filter chip that hid nothing — the
panel reported "212 matches" with every middleware span still listed.

Reading the implementation: spanName is compared with an exact string
equality against operationName, so only "=" and "!=" against a single
literal name do anything. It is also pushed into the tag-filter list,
where "=~"/"!~" are honoured — but that matcher looks the key up among
the span's tags, and a span's name is not a tag, so the pattern matches
nothing and the negative filter passes everything.

So pattern-based exclusion is not available in this panel. Remove the
option rather than leave a chip that suggests filtering is happening, and
say so in the description, pointing at the built-in Critical path button
as the one lever that does narrow the view.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 .../dashboards/obs-distributed-traces.json    | 22 ++-----------------
 1 file changed, 2 insertions(+), 20 deletions(-)

diff --git a/k8s/grafana/dashboards/obs-distributed-traces.json b/k8s/grafana/dashboards/obs-distributed-traces.json
index 4c0b5e6..b578596 100644
--- a/k8s/grafana/dashboards/obs-distributed-traces.json
+++ b/k8s/grafana/dashboards/obs-distributed-traces.json
@@ -170,7 +170,7 @@
     {
       "type": "traces",
       "title": "Trace $traceId",
-      "description": "Grafana's own trace view: every span nested in order across LibreChat, Keycloak, the Kong gateways and the backend. Click a span to expand its attributes; from a span, 'Logs for this span' jumps to Loki.\n\nA default span filter hides the framework plumbing — Express middleware, mongoose/mongodb calls, kong.dns, raw SQL — which is most of a 212-span trace and none of the flow. Clear the filter in the panel, or toggle 'Show all spans', to get them back.\n\nThis is a DISPLAY filter: those spans are still produced and still stored in Tempo.",
+      "description": "Grafana's own trace view: every span nested in order across LibreChat, Keycloak, the Kong gateways and the backend. Click a span to expand its attributes; from a span, 'Logs for this span' jumps to Loki.\n\nThe framework plumbing — Express middleware, mongoose/mongodb, kong.dns, raw SQL — is most of the span count and cannot be hidden from here: this panel's span-name filter is an exact string comparison on operationName, so a pattern matches nothing and every span passes. Only the tag filters take a regex, and a span's name is not a tag. Use the panel's own 'Critical path' button to collapse the view to what determines the duration.\n\nRemoving those spans for real means turning the instrumentations off at the source.",
       "datasource": {
         "type": "tempo",
         "uid": "tempo"
@@ -193,25 +193,7 @@
           "limit": 300
         }
       ],
-      "options": {
-        "spanFilters": {
-          "spanName": "middleware - .*|mongoose\\..*|mongodb\\..*|kong\\.dns|SELECT|INSERT|UPDATE|DELETE|router - .*",
-          "spanNameOperator": "!~",
-          "serviceName": "",
-          "serviceNameOperator": "=",
-          "from": "",
-          "to": "",
-          "tags": [
-            {
-              "id": "1",
-              "operator": "="
-            }
-          ],
-          "matchesOnly": true,
-          "criticalPathOnly": false,
-          "query": ""
-        }
-      }
+      "options": {}
     },
     {
       "type": "row",