The server ships 65 tools by default. An MCP client sends the whole tool
list on every turn, so that was 75 KiB — measured at ~19k prompt tokens
per request before the user has asked anything. A three-turn conversation
therefore tripped Kong's ai-rate-limiting-advanced, 50k tokens per minute
per consumer, and the assistant sat in a 429 retry loop: 15 rejected calls
for one that got through. Proven to be Kong and not the provider —
upstream_status empty, tries empty, latencies.proxy -1, and x-ai-ratelimit
headers on the response.
Down to 17 tools and ~5.4k tokens, a 72% cut, keeping query_prometheus,
query_loki_logs, the label listings and dashboard search.
Dropped: everything Cloud-only on OSS Grafana (incident, oncall, sift,
asserts, pyroscope), every datasource this cluster does not run (athena,
clickhouse, cloudwatch, influxdb, elasticsearch, graphite, quickwit,
snowflake), and the administration surface (provisioning, plugin, api,
config, folder, annotations, snapshot, rendering, navigation).
--disable-write on top: the service account is Viewer, so writes would
fail anyway, and this keeps their definitions out of every request.
Note what this does not fix: list_prometheus_metric_names still returns
all 3330 metric names, 137 KiB, and the assistant called it twice while
exploring. The rate limit did its job by refusing those calls before they
reached the provider — raising it would have paid for the same behaviour.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exposes Prometheus, Loki and Tempo as tools an LLM can call, so a question
like "what failed in the last transaction" is answered by querying rather
than by building another panel — which is where the trace dashboard kept
running out of road.
Works against OSS Grafana; the Cloud-only tools (Incidents, OnCall, Sift,
Agent Observability, Assistant) are simply absent. Verified end to end:
the MCP handshake succeeds and list_datasources returns Loki, Prometheus
and Tempo.
Authenticates with a Grafana service account token, Viewer role, created
once through /api/serviceaccounts and stored in the gitignored
k8s/secret.yaml. secret.example.yaml documents how to mint it, since
Grafana returns the value only at creation.
Two things the deployment needs that are not obvious:
* runAsUser must be numeric. The image declares USER mcp-grafana by
name, Kubernetes cannot resolve that to check it is not root, and
refuses to start the container under runAsNonRoot with "image has
non-numeric user". 1000/999 is what the image actually uses.
* --allowed-hosts must name the Service. The server validates the Host
header against a loopback-only default as DNS-rebinding protection, so
the first in-cluster call returned "forbidden: host not allowed".
Named rather than disabled with "*": the check is worth keeping.
Deliberately in-cluster only, with no OIDC and no audit trail — anything
that can reach this Service reads all the telemetry. The intended end
state is the same treatment as the five domain MCP servers, behind a Kong
gateway with openid-connect, consumer groups and http-log.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland