Stéphane Tailland
ad419851c6
The server ships 65 tools by default. An MCP client sends the whole tool list on every turn, so that was 75 KiB — measured at ~19k prompt tokens per request before the user has asked anything. A three-turn conversation therefore tripped Kong's ai-rate-limiting-advanced, 50k tokens per minute per consumer, and the assistant sat in a 429 retry loop: 15 rejected calls for one that got through. Proven to be Kong and not the provider — upstream_status empty, tries empty, latencies.proxy -1, and x-ai-ratelimit headers on the response. Down to 17 tools and ~5.4k tokens, a 72% cut, keeping query_prometheus, query_loki_logs, the label listings and dashboard search. Dropped: everything Cloud-only on OSS Grafana (incident, oncall, sift, asserts, pyroscope), every datasource this cluster does not run (athena, clickhouse, cloudwatch, influxdb, elasticsearch, graphite, quickwit, snowflake), and the administration surface (provisioning, plugin, api, config, folder, annotations, snapshot, rendering, navigation). --disable-write on top: the service account is Viewer, so writes would fail anyway, and this keeps their definitions out of every request. Note what this does not fix: list_prometheus_metric_names still returns all 3330 metric names, 137 KiB, and the assistant called it twice while exploring. The rate limit did its job by refusing those calls before they reached the provider — raising it would have paid for the same behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
97 lines
4.2 KiB
YAML
97 lines
4.2 KiB
YAML
# Grafana's own MCP server: it exposes Prometheus, Loki and Tempo as tools an
|
|
# LLM can call, so a question like "what failed in the last transaction" is
|
|
# answered by querying, rather than by building another panel.
|
|
#
|
|
# Works against OSS Grafana 9.0+. The Cloud-only tools — Incidents, OnCall,
|
|
# Sift, Agent Observability, Assistant — are simply absent here; the
|
|
# datasource, dashboard and alerting tools are the ones that matter.
|
|
#
|
|
# Reachable in-cluster only for now, with no OIDC and no audit trail. The
|
|
# intended end state is the same treatment as the five domain MCP servers:
|
|
# behind a Kong gateway with openid-connect, consumer groups and http-log.
|
|
# Until then, anything that can reach this Service can read all the telemetry.
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: mcp-grafana
|
|
labels:
|
|
app.kubernetes.io/name: mcp-grafana
|
|
app.kubernetes.io/component: mcp
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: mcp-grafana
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: mcp-grafana
|
|
app.kubernetes.io/component: mcp
|
|
spec:
|
|
containers:
|
|
- name: mcp-grafana
|
|
image: mcp/grafana:latest
|
|
args:
|
|
# Streamable HTTP rather than stdio: this runs as a service other
|
|
# pods connect to, not as a subprocess of one client.
|
|
- "-t"
|
|
- "streamable-http"
|
|
- "--log-level"
|
|
- "info"
|
|
# DNS-rebinding protection: the server rejects any Host header not
|
|
# on this list, defaulting to loopback only — which is why the first
|
|
# in-cluster probe got "forbidden: host not allowed". Name the
|
|
# Service instead of disabling validation with "*": the point of the
|
|
# check is that only callers using the expected name get through.
|
|
- "--allowed-hosts"
|
|
- "mcp-grafana:8000,mcp-grafana.observability.svc.cluster.local:8000"
|
|
# 65 tools by default, 75 KiB of definitions — and an MCP client
|
|
# sends the whole tool list on every turn. Measured: that alone was
|
|
# ~19k prompt tokens per request, which is what made a three-turn
|
|
# conversation trip Kong's 50k-tokens-per-minute AI rate limit and
|
|
# fail with 429 before reaching the provider.
|
|
#
|
|
# Kept: the datasources this stack actually has, plus search to find
|
|
# things by name. Dropped: everything Cloud-only here (incident,
|
|
# oncall, sift, asserts, pyroscope), every datasource that does not
|
|
# exist (athena, clickhouse, cloudwatch, influxdb, elasticsearch,
|
|
# graphite, quickwit, snowflake), and the Grafana-administration
|
|
# surface (provisioning, plugin, api, config, folder, annotations,
|
|
# snapshot, rendering, navigation).
|
|
- "--enabled-tools"
|
|
- "search,datasource,prometheus,loki"
|
|
# The service account is Viewer, so writes would fail anyway —
|
|
# this removes their definitions from what is sent every turn.
|
|
- "--disable-write"
|
|
env:
|
|
- name: GRAFANA_URL
|
|
# In-cluster address, not the ingress: no TLS hop, no dependency
|
|
# on sttlab.local resolving inside the cluster.
|
|
value: http://grafana:3000
|
|
- name: GRAFANA_SERVICE_ACCOUNT_TOKEN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: observability-secrets
|
|
key: GRAFANA_SERVICE_ACCOUNT_TOKEN
|
|
ports:
|
|
- name: http
|
|
containerPort: 8000
|
|
resources:
|
|
requests:
|
|
cpu: 20m
|
|
memory: 64Mi
|
|
limits:
|
|
memory: 256Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
runAsNonRoot: true
|
|
# Numeric, because the image declares its user by name
|
|
# (USER mcp-grafana). Kubernetes cannot resolve a name to check it
|
|
# is not root, and refuses to start the container with
|
|
# runAsNonRoot: "image has non-numeric user". 1000/999 are what the
|
|
# image actually uses.
|
|
runAsUser: 1000
|
|
runAsGroup: 999
|
|
capabilities:
|
|
drop: ["ALL"]
|