k8s/secret.yaml held the Grafana admin credentials and, since the MCP
server landed, its service account token — gitignored, so deploying this
stack required the right file to exist on the right laptop. It now comes
from Vault through External Secrets, and nothing sensitive has to exist
locally.
A dedicated store rather than reusing vault-kong-v2: that one's AppRole
carries kong-v2-readonly, scoped to secrets/data/kong-v2/*, so it cannot
see this path — confirmed with a probe ExternalSecret that failed with
"could not get secret data from provider". Widening that policy would have
handed the Kong reader observability's secrets for no reason. So
observability-readonly and observability-eso were created in Vault,
mirroring the existing pattern exactly.
The AppRole credentials stay out of git; the ClusterSecretStore comments
carry the commands that mint them.
dataFrom.extract pulls the whole path instead of naming each key, so a new
key in Vault reaches the cluster with no edit here. The target keeps the
name every consumer already uses, so neither the Grafana Deployment nor
mcp-grafana changed — verified, neither pod restarted and the token in the
cluster matches Vault byte for byte.
secret.example.yaml no longer describes a file to create; it documents
where the keys live and how the Grafana token is minted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exposes Prometheus, Loki and Tempo as tools an LLM can call, so a question
like "what failed in the last transaction" is answered by querying rather
than by building another panel — which is where the trace dashboard kept
running out of road.
Works against OSS Grafana; the Cloud-only tools (Incidents, OnCall, Sift,
Agent Observability, Assistant) are simply absent. Verified end to end:
the MCP handshake succeeds and list_datasources returns Loki, Prometheus
and Tempo.
Authenticates with a Grafana service account token, Viewer role, created
once through /api/serviceaccounts and stored in the gitignored
k8s/secret.yaml. secret.example.yaml documents how to mint it, since
Grafana returns the value only at creation.
Two things the deployment needs that are not obvious:
* runAsUser must be numeric. The image declares USER mcp-grafana by
name, Kubernetes cannot resolve that to check it is not root, and
refuses to start the container under runAsNonRoot with "image has
non-numeric user". 1000/999 is what the image actually uses.
* --allowed-hosts must name the Service. The server validates the Host
header against a loopback-only default as DNS-rebinding protection, so
the first in-cluster call returned "forbidden: host not allowed".
Named rather than disabled with "*": the check is worth keeping.
Deliberately in-cluster only, with no OIDC and no audit trail — anything
that can reach this Service reads all the telemetry. The intended end
state is the same treatment as the five domain MCP servers, behind a Kong
gateway with openid-connect, consumer groups and http-log.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deploys the whole stack into the observability namespace of the local
cluster with Kustomize alone — no helm install, no dashboard created by
hand in the UI. The Makefile carries every command: deploy, dashboards,
diagnostics, PVC flush, teardown.
Dashboards are files, never UI objects. Each carries a stable uid, which
is what keeps URLs, cross-links and alerts alive across redeployments, and
references datasources by stable uid rather than by name. The dashboard
ConfigMap is generated without a name-suffix hash so applying it updates
it in place and Grafana reloads without a restart, while the provisioning
ConfigMaps keep their hash precisely so that a change there does force one.
Prometheus scrapes the kubelet /metrics/resource endpoint rather than
/metrics/cadvisor: on this cluster cadvisor exposes machine_* series only
and no container_* at all, which would leave every CPU and memory panel
empty.
Alloy ships every pod's logs to Loki through the Kubernetes API, so no
/var/log mount is needed. Tempo's metrics_generator feeds span metrics and
the service graph back into Prometheus, and the datasources are wired so a
log line's trace_id opens the matching trace.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland