Add Grafana's MCP server

Exposes Prometheus, Loki and Tempo as tools an LLM can call, so a question
like "what failed in the last transaction" is answered by querying rather
than by building another panel — which is where the trace dashboard kept
running out of road.

Works against OSS Grafana; the Cloud-only tools (Incidents, OnCall, Sift,
Agent Observability, Assistant) are simply absent. Verified end to end:
the MCP handshake succeeds and list_datasources returns Loki, Prometheus
and Tempo.

Authenticates with a Grafana service account token, Viewer role, created
once through /api/serviceaccounts and stored in the gitignored
k8s/secret.yaml. secret.example.yaml documents how to mint it, since
Grafana returns the value only at creation.

Two things the deployment needs that are not obvious:

  * runAsUser must be numeric. The image declares USER mcp-grafana by
    name, Kubernetes cannot resolve that to check it is not root, and
    refuses to start the container under runAsNonRoot with "image has
    non-numeric user". 1000/999 is what the image actually uses.

  * --allowed-hosts must name the Service. The server validates the Host
    header against a loopback-only default as DNS-rebinding protection, so
    the first in-cluster call returned "forbidden: host not allowed".
    Named rather than disabled with "*": the check is worth keeping.

Deliberately in-cluster only, with no OIDC and no audit trail — anything
that can reach this Service reads all the telemetry. The intended end
state is the same treatment as the five domain MCP servers, behind a Kong
gateway with openid-connect, consumer groups and http-log.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/kustomization.yaml

@@ -11,6 +11,7 @@ resources:
   - alloy
   - grafana
   - grafana/dashboards
+  - mcp-grafana
   - ingress
 
 labels: