Expose only the tools this stack has datasources for

The server ships 65 tools by default. An MCP client sends the whole tool
list on every turn, so that was 75 KiB — measured at ~19k prompt tokens
per request before the user has asked anything. A three-turn conversation
therefore tripped Kong's ai-rate-limiting-advanced, 50k tokens per minute
per consumer, and the assistant sat in a 429 retry loop: 15 rejected calls
for one that got through. Proven to be Kong and not the provider —
upstream_status empty, tries empty, latencies.proxy -1, and x-ai-ratelimit
headers on the response.

Down to 17 tools and ~5.4k tokens, a 72% cut, keeping query_prometheus,
query_loki_logs, the label listings and dashboard search.

Dropped: everything Cloud-only on OSS Grafana (incident, oncall, sift,
asserts, pyroscope), every datasource this cluster does not run (athena,
clickhouse, cloudwatch, influxdb, elasticsearch, graphite, quickwit,
snowflake), and the administration surface (provisioning, plugin, api,
config, folder, annotations, snapshot, rendering, navigation).
--disable-write on top: the service account is Viewer, so writes would
fail anyway, and this keeps their definitions out of every request.

Note what this does not fix: list_prometheus_metric_names still returns
all 3330 metric names, 137 KiB, and the assistant called it twice while
exploring. The rate limit did its job by refusing those calls before they
reached the provider — raising it would have paid for the same behaviour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Stéphane Tailland
2026-08-27 17:45:34 +02:00
parent 6d020572e0
commit ad419851c6
+18
View File
@@ -45,6 +45,24 @@ spec:
# check is that only callers using the expected name get through.
- "--allowed-hosts"
- "mcp-grafana:8000,mcp-grafana.observability.svc.cluster.local:8000"
# 65 tools by default, 75 KiB of definitions — and an MCP client
# sends the whole tool list on every turn. Measured: that alone was
# ~19k prompt tokens per request, which is what made a three-turn
# conversation trip Kong's 50k-tokens-per-minute AI rate limit and
# fail with 429 before reaching the provider.
#
# Kept: the datasources this stack actually has, plus search to find
# things by name. Dropped: everything Cloud-only here (incident,
# oncall, sift, asserts, pyroscope), every datasource that does not
# exist (athena, clickhouse, cloudwatch, influxdb, elasticsearch,
# graphite, quickwit, snowflake), and the Grafana-administration
# surface (provisioning, plugin, api, config, folder, annotations,
# snapshot, rendering, navigation).
- "--enabled-tools"
- "search,datasource,prometheus,loki"
# The service account is Viewer, so writes would fail anyway —
# this removes their definitions from what is sent every turn.
- "--disable-write"
env:
- name: GRAFANA_URL
# In-cluster address, not the ingress: no TLS hop, no dependency