Add a traces dashboard and make the span correlations work

Trace search lived only in Explore. `obs-traces` brings it into a dashboard:
TraceQL assembled from a service selector, a minimum duration, a status and a
free-text field, plus the aggregate RED view from the span metrics Tempo derives
from the traces themselves.

The status and free-text variables hold TraceQL fragments rather than values, so
an empty one contributes nothing and the query stays valid — the same reason the
log dashboards inject their search term the way they do.

Both span correlations were declared but non-functional.

tracesToLogsV2 mapped service.name onto Loki's `app` label, which Alloy derives
from app.kubernetes.io/name. That label exists for Keycloak and the Classic
Models API but not for the six Kong gateways or LibreChat — and the gateways are
the only place the trace id actually appears in the logs, so the mapping pointed
squarely at the services where it could not resolve. Replaced with a custom
query on ${__trace.traceId}, which matches any source writing the id anywhere in
the line.

tracesToMetrics had no queries at all and rendered a bare datasource link. It
now carries rate, error-rate and p95 against traces_spanmetrics_*, scoped to the
span's own service.

The service graph stays a table: serviceMap is a query type Grafana's frontend
assembles for Explore and returns no frames through the dashboard query API.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/grafana/provisioning/datasources.yaml

@@ -64,19 +64,42 @@ datasources:
         datasourceUid: prometheus
       nodeGraph:
         enabled: true
+      # Span -> logs by trace id rather than by service name.
+      #
+      # The service-name mapping this replaces resolved `service.name` to Loki's
+      # `app` label, which Alloy derives from app.kubernetes.io/name. That label
+      # exists for keycloak and classic-models-api but not for the six Kong
+      # gateways or librechat — and the gateways are precisely where the trace
+      # id does appear in the logs. The mapping pointed at the services where it
+      # could not work.
+      #
+      # A custom query on ${__trace.traceId} sidesteps the label entirely: every
+      # source that writes the id anywhere in the line is matched. The stream
+      # selector is spelled out rather than `.+` so the query stays bounded.
       tracesToLogsV2:
         datasourceUid: loki
-        spanStartTimeShift: -5m
-        spanEndTimeShift: 5m
-        filterByTraceID: true
+        spanStartTimeShift: -1h
+        spanEndTimeShift: 1h
+        filterByTraceID: false
         filterBySpanID: false
-        tags:
-          - key: service.name
-            value: app
+        customQuery: true
+        query: '{job=~"kong-audit|keycloak/keycloak|classic-models/classic-models-api|librechat/librechat"} |= "$${__trace.traceId}"'
+      # Span -> metrics against the span metrics Tempo's own metrics_generator
+      # feeds into Prometheus. Without explicit queries this block renders a
+      # bare datasource link and nothing else, which is what it did before.
+      # $$__tags expands to the tag mapping below, so each query is scoped to
+      # the span's own service.
       tracesToMetrics:
         datasourceUid: prometheus
-        spanStartTimeShift: -5m
-        spanEndTimeShift: 5m
+        spanStartTimeShift: -30m
+        spanEndTimeShift: 30m
         tags:
           - key: service.name
             value: service
+        queries:
+          - name: 'Request rate'
+            query: 'sum by (span_name) (rate(traces_spanmetrics_calls_total{$$__tags}[$$__rate_interval]))'
+          - name: 'Error rate'
+            query: 'sum by (span_name) (rate(traces_spanmetrics_calls_total{$$__tags,status_code="STATUS_CODE_ERROR"}[$$__rate_interval]))'
+          - name: 'Latency p95'
+            query: 'histogram_quantile(0.95, sum by (span_name, le) (rate(traces_spanmetrics_latency_bucket{$$__tags}[$$__rate_interval])))'