Take the observability secrets from Vault

k8s/secret.yaml held the Grafana admin credentials and, since the MCP
server landed, its service account token — gitignored, so deploying this
stack required the right file to exist on the right laptop. It now comes
from Vault through External Secrets, and nothing sensitive has to exist
locally.

A dedicated store rather than reusing vault-kong-v2: that one's AppRole
carries kong-v2-readonly, scoped to secrets/data/kong-v2/*, so it cannot
see this path — confirmed with a probe ExternalSecret that failed with
"could not get secret data from provider". Widening that policy would have
handed the Kong reader observability's secrets for no reason. So
observability-readonly and observability-eso were created in Vault,
mirroring the existing pattern exactly.

The AppRole credentials stay out of git; the ClusterSecretStore comments
carry the commands that mint them.

dataFrom.extract pulls the whole path instead of naming each key, so a new
key in Vault reaches the cluster with no edit here. The target keeps the
name every consumer already uses, so neither the Grafana Deployment nor
mcp-grafana changed — verified, neither pod restarted and the token in the
cluster matches Vault byte for byte.

secret.example.yaml no longer describes a file to create; it documents
where the keys live and how the Grafana token is minted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/secret.example.yaml

@@ -1,17 +1,14 @@
-# Template: copy to secret.yaml (gitignored) and fill in the values.
-#   make secret
-apiVersion: v1
-kind: Secret
-metadata:
-  name: observability-secrets
-  namespace: observability
-type: Opaque
-stringData:
-  GF_SECURITY_ADMIN_USER: admin
-  GF_SECURITY_ADMIN_PASSWORD: <CHANGE_ME>
-  # Token of a Grafana service account, created once with the admin credentials:
-  #   POST /api/serviceaccounts        {"name":"mcp-grafana","role":"Viewer"}
-  #   POST /api/serviceaccounts/<id>/tokens  {"name":"mcp-grafana-token"}
-  # Grafana returns the value once, at creation. Viewer is enough to query the
-  # datasources and list dashboards; raise it only if a tool needs writes.
-  GRAFANA_SERVICE_ACCOUNT_TOKEN: glsa_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+# NO LONGER APPLIED. observability-secrets now comes from Vault through
+# External Secrets — see k8s/secrets/. This file is kept only to document
+# which keys the stack expects and where they live.
+#
+# Vault path: secrets/observability/grafana
+#   GF_SECURITY_ADMIN_USER
+#   GF_SECURITY_ADMIN_PASSWORD
+#   GRAFANA_SERVICE_ACCOUNT_TOKEN   token of the 'mcp-grafana' service
+#     account, Viewer role. Grafana returns it once, at creation:
+#       POST /api/serviceaccounts             {"name":"mcp-grafana","role":"Viewer"}
+#       POST /api/serviceaccounts/<id>/tokens {"name":"mcp-grafana-token"}
+#
+# The ExternalSecret pulls the whole path, so adding a key in Vault reaches
+# the cluster without editing anything here.