Expose only the tools this stack has datasources for

The server ships 65 tools by default. An MCP client sends the whole tool
list on every turn, so that was 75 KiB — measured at ~19k prompt tokens
per request before the user has asked anything. A three-turn conversation
therefore tripped Kong's ai-rate-limiting-advanced, 50k tokens per minute
per consumer, and the assistant sat in a 429 retry loop: 15 rejected calls
for one that got through. Proven to be Kong and not the provider —
upstream_status empty, tries empty, latencies.proxy -1, and x-ai-ratelimit
headers on the response.

Down to 17 tools and ~5.4k tokens, a 72% cut, keeping query_prometheus,
query_loki_logs, the label listings and dashboard search.

Dropped: everything Cloud-only on OSS Grafana (incident, oncall, sift,
asserts, pyroscope), every datasource this cluster does not run (athena,
clickhouse, cloudwatch, influxdb, elasticsearch, graphite, quickwit,
snowflake), and the administration surface (provisioning, plugin, api,
config, folder, annotations, snapshot, rendering, navigation).
--disable-write on top: the service account is Viewer, so writes would
fail anyway, and this keeps their definitions out of every request.

Note what this does not fix: list_prometheus_metric_names still returns
all 3330 metric names, 137 KiB, and the assistant called it twice while
exploring. The rate limit did its job by refusing those calls before they
reached the provider — raising it would have paid for the same behaviour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/grafana/dashboards/obs-distributed-traces.json

@@ -1,7 +1,7 @@
 {
   "uid": "obs-distributed-traces",
   "title": "Distributed Traces",
-  "description": "Find a trace, and keep only the spans worth looking at. Built because Explore's unfiltered search is unusable here: Tempo does not return {} newest-first, so health checks crowd out real transactions.\n\nDeliberately carries no metric panels. A span's own duration is in the span tables, measured; aggregate RED by span name is what the per-application dashboards already show, and neither answers 'the metrics of this span'.",
+  "description": "Follow one transaction end to end: pick it, read every span in order, see which tokens were issued, and read what was exchanged at each hop.",
   "tags": [
     "observability",
     "tracing",
@@ -16,34 +16,43 @@
     "to": "now"
   },
   "editable": true,
-  "graphTooltip": 1,
+  "graphTooltip": 0,
   "templating": {
     "list": [
       {
-        "name": "service",
+        "name": "traceFilter",
-        "label": "Service",
+        "label": "Show",
-        "type": "query",
+        "type": "custom",
-        "datasource": {
+        "query": "LibreChat transactions : {trace:rootService=\"librechat\"} && {resource.service.name!=\"librechat\"},Reaching a Kong gateway : {trace:rootService=\"librechat\"} && {name=~\"kong\\\\..*\"},All LibreChat traces (incl. /metrics scrapes) : {trace:rootService=\"librechat\"},Everything (very noisy) : {}",
-          "type": "prometheus",
+        "options": [
-          "uid": "prometheus"
+          {
+            "text": "LibreChat transactions",
+            "value": "{trace:rootService=\"librechat\"} && {resource.service.name!=\"librechat\"}",
+            "selected": true
           },
-        "query": {
+          {
-          "query": "label_values(traces_spanmetrics_calls_total, service)",
+            "text": "Reaching a Kong gateway",
-          "refId": "service"
+            "value": "{trace:rootService=\"librechat\"} && {name=~\"kong\\\\..*\"}",
+            "selected": false
           },
-        "refresh": 2,
+          {
-        "multi": true,
+            "text": "All LibreChat traces (incl. /metrics scrapes)",
-        "includeAll": true,
+            "value": "{trace:rootService=\"librechat\"}",
-        "allValue": ".+",
+            "selected": false
-        "current": {
+          },
-          "text": [
+          {
-            "All"
+            "text": "Everything (very noisy)",
+            "value": "{}",
+            "selected": false
+          }
         ],
-          "value": [
+        "current": {
-            "$__all"
+          "text": "LibreChat transactions",
-          ]
+          "value": "{trace:rootService=\"librechat\"} && {resource.service.name!=\"librechat\"}"
         },
-        "description": "Services known to Tempo's span metrics. Drives the trace, span and metric panels."
+        "includeAll": false,
+        "multi": false,
+        "description": "Traces started BY LibreChat and leaving it. Rooting on LibreChat is what makes this a user-facing transaction rather than infrastructure chatter; requiring a second service is what drops Prometheus scraping LibreChat's own /metrics, which otherwise fills the list."
       },
       {
         "name": "traceId",
@@ -54,14 +63,25 @@
           "text": "",
           "value": ""
         },
-        "description": "Set by clicking a Trace ID in the top table, or paste one. Empty means the log filter matches everything."
+        "description": "Set by clicking a Trace ID above, or paste one."
+      },
+      {
+        "name": "userId",
+        "label": "User ID",
+        "type": "textbox",
+        "query": "",
+        "current": {
+          "text": "",
+          "value": ""
+        },
+        "description": "Set by clicking an identity in the panel below. Empty means the Keycloak panel is scoped only by time."
       }
     ]
   },
   "panels": [
     {
       "type": "row",
-      "title": "Traces",
+      "title": "1 — Pick a transaction",
       "gridPos": {
         "h": 1,
         "w": 24,
@@ -73,14 +93,14 @@
     },
     {
       "type": "table",
-      "title": "Traces on the dashboard time range",
+      "title": "Transactions",
-      "description": "TraceQL over the dashboard's time range. Filtered by service so the health-check noise stays out: an unfiltered {} query is NOT returned newest-first by Tempo and the limit truncates arbitrarily, which is why browsing raw Explore misses recent traces. Click a Trace ID to load it below.",
+      "description": "Traces LibreChat started that reached another service — the user-facing transactions.\n\nCounting spans was the previous approach and it let LibreChat's own /metrics scrapes through: with 50 of them in six hours they crowded out the two real conversations. Rooting on the service that starts a transaction is both simpler and stricter.\n\nThe trade: anything not entering through LibreChat is invisible here — a direct call to a gateway, or a curl against an API. Use 'Everything' for those.\n\nClick a Trace ID to load it below.",
       "datasource": {
         "type": "tempo",
         "uid": "tempo"
       },
       "gridPos": {
-        "h": 9,
+        "h": 8,
         "w": 24,
         "x": 0,
         "y": 1
@@ -93,10 +113,9 @@
             "uid": "tempo"
           },
           "queryType": "traceql",
-          "query": "{resource.service.name=~\"$service\"}",
+          "query": "$traceFilter",
           "tableType": "traces",
-          "limit": 50,
+          "limit": 50
-          "spss": 20
         }
       ],
       "fieldConfig": {
@@ -107,18 +126,6 @@
           }
         },
         "overrides": [
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "traceIdHidden"
-            },
-            "properties": [
-              {
-                "id": "custom.hidden",
-                "value": true
-              }
-            ]
-          },
           {
             "matcher": {
               "id": "byName",
@@ -129,8 +136,8 @@
                 "id": "links",
                 "value": [
                   {
-                    "title": "Load this trace below",
+                    "title": "Open this trace below",
-                    "url": "d/obs-distributed-traces/distributed-traces?${__url_time_range}&var-traceId=${__value.raw}&var-service=${service:queryparam}"
+                    "url": "d/obs-distributed-traces/distributed-traces?${__url_time_range}&var-traceId=${__value.raw}"
                   }
                 ]
               }
@@ -150,30 +157,30 @@
     },
     {
       "type": "row",
-      "title": "Spans that matter",
+      "title": "2 — Every span of the trace, in order",
       "gridPos": {
         "h": 1,
         "w": 24,
         "x": 0,
+        "y": 9
+      },
+      "collapsed": false,
+      "panels": []
+    },
+    {
+      "type": "traces",
+      "title": "Trace $traceId",
+      "description": "Grafana's own trace view: every span nested in order across LibreChat, Keycloak, the Kong gateways and the backend. Click a span to expand its attributes; from a span, 'Logs for this span' jumps to Loki.\n\nThe framework plumbing — Express middleware, mongoose/mongodb, kong.dns, raw SQL — is most of the span count and cannot be hidden from here: this panel's span-name filter is an exact string comparison on operationName, so a pattern matches nothing and every span passes. Only the tag filters take a regex, and a span's name is not a tag. Use the panel's own 'Critical path' button to collapse the view to what determines the duration.\n\nRemoving those spans for real means turning the instrumentations off at the source.",
+      "datasource": {
+        "type": "tempo",
+        "uid": "tempo"
+      },
+      "gridPos": {
+        "h": 22,
+        "w": 24,
+        "x": 0,
         "y": 10
       },
-      "collapsed": false,
-      "panels": []
-    },
-    {
-      "type": "table",
-      "title": "HTTP hops between components",
-      "description": "One row per component entry: these are the SERVER-kind spans, i.e. the inbound HTTP request each service received. Browser to LibreChat, LibreChat to the llm gateway, gateway to a domain gateway, gateway to the backend. Everything else — express middleware, DNS, SQL — is deliberately excluded.",
-      "datasource": {
-        "type": "tempo",
-        "uid": "tempo"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 12,
-        "x": 0,
-        "y": 11
-      },
       "targets": [
         {
           "refId": "A",
@@ -182,82 +189,141 @@
             "uid": "tempo"
           },
           "queryType": "traceql",
-          "query": "{resource.service.name=~\"$service\" && kind=server} | select(span.http.method, span.http.status_code, span.http.route, span.http.url)",
+          "query": "$traceId",
-          "tableType": "spans",
+          "limit": 300
-          "limit": 50,
-          "spss": 20
         }
       ],
-      "fieldConfig": {
+      "options": {}
-        "defaults": {
-          "custom": {
-            "align": "auto",
-            "filterable": true
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "showHeader": true
-      }
-    },
-    {
-      "type": "table",
-      "title": "Kong plugin execution",
-      "description": "Every plugin Kong ran, with its own duration — openid-connect, rate-limiting, request-transformer, opentelemetry, and so on. This is what shows which plugin is costing latency on a gateway hop.",
-      "datasource": {
-        "type": "tempo",
-        "uid": "tempo"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 12,
-        "x": 12,
-        "y": 11
-      },
-      "targets": [
-        {
-          "refId": "A",
-          "datasource": {
-            "type": "tempo",
-            "uid": "tempo"
-          },
-          "queryType": "traceql",
-          "query": "{resource.service.name=~\"$service\" && name=~\"kong\\\\..*plugin.*\"}",
-          "tableType": "spans",
-          "limit": 50,
-          "spss": 20
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "align": "auto",
-            "filterable": true
-          }
-        },
-        "overrides": []
-      },
-      "options": {
-        "showHeader": true
-      }
     },
     {
       "type": "row",
-      "title": "Logs for the selected trace",
+      "title": "3 — Identity: what was issued, to whom, for what",
       "gridPos": {
         "h": 1,
         "w": 24,
         "x": 0,
-        "y": 21
+        "y": 32
       },
       "collapsed": false,
       "panels": []
     },
     {
-      "type": "logs",
+      "type": "table",
-      "title": "Logs carrying trace $traceId",
+      "title": "Identities authenticated in this trace",
-      "description": "Correlation is by TRACE id, not span id: none of the emitters log a span id, so this is every log line of the whole trace rather than of one span.\n\nCoverage is uneven and worth knowing: kong-audit and the backend's access log carry the id on every line; the backend's application log does too since v1.0.4. LibreChat logs carry no trace id at all, so its lines never appear here.\n\nWith traceId empty the filter matches everything — paste an id or click one above.",
+      "description": "Who Kong authenticated at each hop of the selected trace, read from the audit records' authenticated_entity.id. A transaction usually carries more than one: LibreChat's own identity on the llm hop, the end user's on the domain hops.\n\nClick a User ID to scope the Keycloak panel below to it. That click is the join Kong does not make for us — its openid-connect plugin calls Keycloak through its own HTTP client, which Kong's propagation module does not touch, so those token exchanges carry their own trace id. The user id is the same on both sides, which is what makes the join possible.",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 33
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "editorMode": "code",
+          "queryType": "range",
+          "expr": "{job=\"kong-audit\"} |= \"$traceId\" | json authuser=\"authenticated_entity.id\", svc=\"service.name\", uri=\"request.uri\", status=\"response.status\""
+        }
+      ],
+      "transformations": [
+        {
+          "id": "extractFields",
+          "options": {
+            "source": "labels",
+            "replace": false
+          }
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "labels": true,
+              "Line": true,
+              "tsNs": true,
+              "id": true
+            },
+            "indexByName": {
+              "Time": 0,
+              "authuser": 1,
+              "svc": 2,
+              "uri": 3,
+              "status": 4
+            },
+            "renameByName": {
+              "authuser": "User ID",
+              "svc": "Gateway service",
+              "uri": "Path",
+              "status": "Status"
+            }
+          }
+        },
+        {
+          "id": "groupBy",
+          "options": {
+            "fields": {
+              "User ID": {
+                "aggregations": [],
+                "operation": "groupby"
+              },
+              "Gateway service": {
+                "aggregations": [],
+                "operation": "groupby"
+              },
+              "Path": {
+                "aggregations": [],
+                "operation": "groupby"
+              },
+              "Status": {
+                "aggregations": [],
+                "operation": "groupby"
+              }
+            }
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "align": "auto",
+            "filterable": true
+          }
+        },
+        "overrides": [
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "User ID"
+            },
+            "properties": [
+              {
+                "id": "links",
+                "value": [
+                  {
+                    "title": "Scope the Keycloak panel to this identity",
+                    "url": "d/obs-distributed-traces/distributed-traces?${__url_time_range}&var-traceId=${traceId}&var-userId=${__value.raw}"
+                  }
+                ]
+              }
+            ]
+          }
+        ]
+      },
+      "options": {
+        "showHeader": true
+      }
+    },
+    {
+      "type": "table",
+      "title": "Keycloak identity events",
+      "description": "Parsed from Keycloak's own event log: token exchanges, logins and code-to-token swaps, with the audience minted, the requesting client, the end user and the grant type. This is where the OIDC substance lives — the Authorization header is REDACTED in the Kong audit records, so claims cannot be read from there, and the raw token is deliberately not logged anywhere.\n\nScoped by USER, not by trace, because Kong does not propagate the trace when its openid-connect plugin calls Keycloak — verified: a request traced end to end produced a TOKEN_EXCHANGE that Keycloak recorded under its own trace id. Click a User ID in the panel above to fill it in; with it empty this shows every identity event in the time range.",
       "datasource": {
         "type": "loki",
         "uid": "loki"
@@ -266,7 +332,102 @@
         "h": 11,
         "w": 24,
         "x": 0,
-        "y": 22
+        "y": 41
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": {
+            "type": "loki",
+            "uid": "loki"
+          },
+          "editorMode": "code",
+          "queryType": "range",
+          "expr": "{job=\"keycloak/keycloak\"} |= \"org.keycloak.events\" |= \"$userId\" | regexp `type=\\\\\"(?P<event>[^\\\\\"]*)\\\\\"` | regexp `clientId=\\\\\"(?P<client>[^\\\\\"]*)\\\\\"` | regexp `audience=\\\\\"(?P<audience>[^\\\\\"]*)\\\\\"` | regexp `username=\\\\\"(?P<user>[^\\\\\"]*)\\\\\"` | regexp `grant_type=\\\\\"(?P<grant>[^\\\\\"]*)\\\\\"` | regexp `scope=\\\\\"(?P<scope>[^\\\\\"]*)\\\\\"`"
+        }
+      ],
+      "transformations": [
+        {
+          "id": "extractFields",
+          "options": {
+            "source": "labels",
+            "replace": false
+          }
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "labels": true,
+              "Line": true,
+              "tsNs": true,
+              "id": true,
+              "Time": false
+            },
+            "indexByName": {
+              "Time": 0,
+              "event": 1,
+              "client": 2,
+              "audience": 3,
+              "user": 4,
+              "grant": 5,
+              "scope": 6
+            },
+            "renameByName": {
+              "event": "Event",
+              "client": "Client",
+              "audience": "Audience",
+              "user": "User",
+              "grant": "Grant type",
+              "scope": "Scopes"
+            }
+          }
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "align": "auto",
+            "filterable": true
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "showHeader": true,
+        "sortBy": [
+          {
+            "displayName": "Time",
+            "desc": true
+          }
+        ]
+      }
+    },
+    {
+      "type": "row",
+      "title": "4 — What was actually exchanged",
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 52
+      },
+      "collapsed": false,
+      "panels": []
+    },
+    {
+      "type": "logs",
+      "title": "Logs of trace $traceId",
+      "description": "Every log line carrying this trace id. The Kong audit records are the substance: request headers, response status, per-stage latencies, and on the domain gateways request_body and response_body — so the MCP tools/call payload and its result are readable in full. Expand a line to read it as fields.\n\nOne gap, and it is a capture gap rather than a display one: the llm gateway has no body capture, so LLM prompts and completions are not recorded.",
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "gridPos": {
+        "h": 14,
+        "w": 24,
+        "x": 0,
+        "y": 53
       },
       "targets": [
         {
@@ -282,7 +443,8 @@
         "showTime": true,
         "wrapLogMessage": true,
         "sortOrder": "Ascending",
-        "enableLogDetails": true
+        "enableLogDetails": true,
+        "prettifyLogMessage": true
       }
     }
   ]

k8s/kustomization.yaml

@@ -4,13 +4,16 @@ namespace: observability
 
 resources:
   - namespace.yaml
-  - secret.yaml
+  # observability-secrets now comes from Vault through External Secrets;
+  # k8s/secret.yaml is no longer applied and can be deleted.
+  - secrets
   - prometheus
   - loki
   - tempo
   - alloy
   - grafana
   - grafana/dashboards
+  - mcp-grafana
   - ingress
 
 labels:

k8s/mcp-grafana/deployment.yaml

@@ -0,0 +1,96 @@
+# Grafana's own MCP server: it exposes Prometheus, Loki and Tempo as tools an
+# LLM can call, so a question like "what failed in the last transaction" is
+# answered by querying, rather than by building another panel.
+#
+# Works against OSS Grafana 9.0+. The Cloud-only tools — Incidents, OnCall,
+# Sift, Agent Observability, Assistant — are simply absent here; the
+# datasource, dashboard and alerting tools are the ones that matter.
+#
+# Reachable in-cluster only for now, with no OIDC and no audit trail. The
+# intended end state is the same treatment as the five domain MCP servers:
+# behind a Kong gateway with openid-connect, consumer groups and http-log.
+# Until then, anything that can reach this Service can read all the telemetry.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mcp-grafana
+  labels:
+    app.kubernetes.io/name: mcp-grafana
+    app.kubernetes.io/component: mcp
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: mcp-grafana
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mcp-grafana
+        app.kubernetes.io/component: mcp
+    spec:
+      containers:
+        - name: mcp-grafana
+          image: mcp/grafana:latest
+          args:
+            # Streamable HTTP rather than stdio: this runs as a service other
+            # pods connect to, not as a subprocess of one client.
+            - "-t"
+            - "streamable-http"
+            - "--log-level"
+            - "info"
+            # DNS-rebinding protection: the server rejects any Host header not
+            # on this list, defaulting to loopback only — which is why the first
+            # in-cluster probe got "forbidden: host not allowed". Name the
+            # Service instead of disabling validation with "*": the point of the
+            # check is that only callers using the expected name get through.
+            - "--allowed-hosts"
+            - "mcp-grafana:8000,mcp-grafana.observability.svc.cluster.local:8000"
+            # 65 tools by default, 75 KiB of definitions — and an MCP client
+            # sends the whole tool list on every turn. Measured: that alone was
+            # ~19k prompt tokens per request, which is what made a three-turn
+            # conversation trip Kong's 50k-tokens-per-minute AI rate limit and
+            # fail with 429 before reaching the provider.
+            #
+            # Kept: the datasources this stack actually has, plus search to find
+            # things by name. Dropped: everything Cloud-only here (incident,
+            # oncall, sift, asserts, pyroscope), every datasource that does not
+            # exist (athena, clickhouse, cloudwatch, influxdb, elasticsearch,
+            # graphite, quickwit, snowflake), and the Grafana-administration
+            # surface (provisioning, plugin, api, config, folder, annotations,
+            # snapshot, rendering, navigation).
+            - "--enabled-tools"
+            - "search,datasource,prometheus,loki"
+            # The service account is Viewer, so writes would fail anyway —
+            # this removes their definitions from what is sent every turn.
+            - "--disable-write"
+          env:
+            - name: GRAFANA_URL
+              # In-cluster address, not the ingress: no TLS hop, no dependency
+              # on sttlab.local resolving inside the cluster.
+              value: http://grafana:3000
+            - name: GRAFANA_SERVICE_ACCOUNT_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: observability-secrets
+                  key: GRAFANA_SERVICE_ACCOUNT_TOKEN
+          ports:
+            - name: http
+              containerPort: 8000
+          resources:
+            requests:
+              cpu: 20m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            runAsNonRoot: true
+            # Numeric, because the image declares its user by name
+            # (USER mcp-grafana). Kubernetes cannot resolve a name to check it
+            # is not root, and refuses to start the container with
+            # runAsNonRoot: "image has non-numeric user". 1000/999 are what the
+            # image actually uses.
+            runAsUser: 1000
+            runAsGroup: 999
+            capabilities:
+              drop: ["ALL"]

k8s/mcp-grafana/kustomization.yaml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - deployment.yaml
+  - service.yaml

k8s/mcp-grafana/service.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: mcp-grafana
+  labels:
+    app.kubernetes.io/name: mcp-grafana
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: mcp-grafana
+  ports:
+    - name: http
+      port: 8000
+      targetPort: http

k8s/secret.example.yaml

@@ -1,11 +1,14 @@
-# Template: copy to secret.yaml (gitignored) and fill in the values.
+# NO LONGER APPLIED. observability-secrets now comes from Vault through
-#   make secret
+# External Secrets — see k8s/secrets/. This file is kept only to document
-apiVersion: v1
+# which keys the stack expects and where they live.
-kind: Secret
+#
-metadata:
+# Vault path: secrets/observability/grafana
-  name: observability-secrets
+#   GF_SECURITY_ADMIN_USER
-  namespace: observability
+#   GF_SECURITY_ADMIN_PASSWORD
-type: Opaque
+#   GRAFANA_SERVICE_ACCOUNT_TOKEN   token of the 'mcp-grafana' service
-stringData:
+#     account, Viewer role. Grafana returns it once, at creation:
-  GF_SECURITY_ADMIN_USER: admin
+#       POST /api/serviceaccounts             {"name":"mcp-grafana","role":"Viewer"}
-  GF_SECURITY_ADMIN_PASSWORD: <CHANGE_ME>
+#       POST /api/serviceaccounts/<id>/tokens {"name":"mcp-grafana-token"}
+#
+# The ExternalSecret pulls the whole path, so adding a key in Vault reaches
+# the cluster without editing anything here.

k8s/secrets/clustersecretstore.yaml

@@ -0,0 +1,35 @@
+# Reads secrets/observability/* from Vault. Deliberately not reusing the
+# vault-kong-v2 store: its AppRole carries the kong-v2-readonly policy, scoped
+# to secrets/data/kong-v2/*, so it simply cannot see this path — verified, a
+# probe ExternalSecret against it failed with "could not get secret data from
+# provider". Widening that policy would have given the Kong reader access to
+# observability's secrets for no reason.
+#
+# The AppRole credentials are NOT in git. Created once against Vault:
+#   vault policy write observability-readonly  (read on secrets/{data,metadata}/observability/*)
+#   vault write auth/approle/role/observability-eso token_policies=observability-readonly
+#   vault read  auth/approle/role/observability-eso/role-id
+#   vault write -f auth/approle/role/observability-eso/secret-id
+#   kubectl -n external-secrets create secret generic vault-approle-observability \
+#     --from-literal=roleId=<...> --from-literal=secretId=<...>
+apiVersion: external-secrets.io/v1
+kind: ClusterSecretStore
+metadata:
+  name: vault-observability
+spec:
+  provider:
+    vault:
+      server: https://vault.sttlab.eu
+      path: secrets
+      version: v2
+      auth:
+        appRole:
+          path: approle
+          roleRef:
+            name: vault-approle-observability
+            namespace: external-secrets
+            key: roleId
+          secretRef:
+            name: vault-approle-observability
+            namespace: external-secrets
+            key: secretId

k8s/secrets/externalsecret.yaml

@@ -0,0 +1,23 @@
+# Replaces the gitignored k8s/secret.yaml: the Grafana admin credentials and
+# the service account token the MCP server authenticates with now come from
+# Vault, so nothing sensitive has to exist on a laptop to deploy this stack.
+#
+# The target keeps the name every consumer already uses, so neither the Grafana
+# Deployment nor mcp-grafana changes.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: observability-secrets
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: vault-observability
+  target:
+    name: observability-secrets
+    creationPolicy: Owner
+  dataFrom:
+    # Whole-secret pull rather than key-by-key: adding a key in Vault then
+    # reaches the cluster without editing this file.
+    - extract:
+        key: observability/grafana

k8s/secrets/kustomization.yaml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - clustersecretstore.yaml
+  - externalsecret.yaml