Take the observability secrets from Vault

k8s/secret.yaml held the Grafana admin credentials and, since the MCP
server landed, its service account token — gitignored, so deploying this
stack required the right file to exist on the right laptop. It now comes
from Vault through External Secrets, and nothing sensitive has to exist
locally.

A dedicated store rather than reusing vault-kong-v2: that one's AppRole
carries kong-v2-readonly, scoped to secrets/data/kong-v2/*, so it cannot
see this path — confirmed with a probe ExternalSecret that failed with
"could not get secret data from provider". Widening that policy would have
handed the Kong reader observability's secrets for no reason. So
observability-readonly and observability-eso were created in Vault,
mirroring the existing pattern exactly.

The AppRole credentials stay out of git; the ClusterSecretStore comments
carry the commands that mint them.

dataFrom.extract pulls the whole path instead of naming each key, so a new
key in Vault reaches the cluster with no edit here. The target keeps the
name every consumer already uses, so neither the Grafana Deployment nor
mcp-grafana changed — verified, neither pod restarted and the token in the
cluster matches Vault byte for byte.

secret.example.yaml no longer describes a file to create; it documents
where the keys live and how the Grafana token is minted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/kustomization.yaml

@@ -4,7 +4,9 @@ namespace: observability
 
 resources:
   - namespace.yaml
-  - secret.yaml
+  # observability-secrets now comes from Vault through External Secrets;
+  # k8s/secret.yaml is no longer applied and can be deleted.
+  - secrets
   - prometheus
   - loki
   - tempo

k8s/secret.example.yaml

@@ -1,17 +1,14 @@
-# Template: copy to secret.yaml (gitignored) and fill in the values.
+# NO LONGER APPLIED. observability-secrets now comes from Vault through
-#   make secret
+# External Secrets — see k8s/secrets/. This file is kept only to document
-apiVersion: v1
+# which keys the stack expects and where they live.
-kind: Secret
+#
-metadata:
+# Vault path: secrets/observability/grafana
-  name: observability-secrets
+#   GF_SECURITY_ADMIN_USER
-  namespace: observability
+#   GF_SECURITY_ADMIN_PASSWORD
-type: Opaque
+#   GRAFANA_SERVICE_ACCOUNT_TOKEN   token of the 'mcp-grafana' service
-stringData:
+#     account, Viewer role. Grafana returns it once, at creation:
-  GF_SECURITY_ADMIN_USER: admin
+#       POST /api/serviceaccounts             {"name":"mcp-grafana","role":"Viewer"}
-  GF_SECURITY_ADMIN_PASSWORD: <CHANGE_ME>
+#       POST /api/serviceaccounts/<id>/tokens {"name":"mcp-grafana-token"}
-  # Token of a Grafana service account, created once with the admin credentials:
+#
-  #   POST /api/serviceaccounts        {"name":"mcp-grafana","role":"Viewer"}
+# The ExternalSecret pulls the whole path, so adding a key in Vault reaches
-  #   POST /api/serviceaccounts/<id>/tokens  {"name":"mcp-grafana-token"}
+# the cluster without editing anything here.
-  # Grafana returns the value once, at creation. Viewer is enough to query the
-  # datasources and list dashboards; raise it only if a tool needs writes.
-  GRAFANA_SERVICE_ACCOUNT_TOKEN: glsa_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

k8s/secrets/clustersecretstore.yaml

@@ -0,0 +1,35 @@
+# Reads secrets/observability/* from Vault. Deliberately not reusing the
+# vault-kong-v2 store: its AppRole carries the kong-v2-readonly policy, scoped
+# to secrets/data/kong-v2/*, so it simply cannot see this path — verified, a
+# probe ExternalSecret against it failed with "could not get secret data from
+# provider". Widening that policy would have given the Kong reader access to
+# observability's secrets for no reason.
+#
+# The AppRole credentials are NOT in git. Created once against Vault:
+#   vault policy write observability-readonly  (read on secrets/{data,metadata}/observability/*)
+#   vault write auth/approle/role/observability-eso token_policies=observability-readonly
+#   vault read  auth/approle/role/observability-eso/role-id
+#   vault write -f auth/approle/role/observability-eso/secret-id
+#   kubectl -n external-secrets create secret generic vault-approle-observability \
+#     --from-literal=roleId=<...> --from-literal=secretId=<...>
+apiVersion: external-secrets.io/v1
+kind: ClusterSecretStore
+metadata:
+  name: vault-observability
+spec:
+  provider:
+    vault:
+      server: https://vault.sttlab.eu
+      path: secrets
+      version: v2
+      auth:
+        appRole:
+          path: approle
+          roleRef:
+            name: vault-approle-observability
+            namespace: external-secrets
+            key: roleId
+          secretRef:
+            name: vault-approle-observability
+            namespace: external-secrets
+            key: secretId

k8s/secrets/externalsecret.yaml

@@ -0,0 +1,23 @@
+# Replaces the gitignored k8s/secret.yaml: the Grafana admin credentials and
+# the service account token the MCP server authenticates with now come from
+# Vault, so nothing sensitive has to exist on a laptop to deploy this stack.
+#
+# The target keeps the name every consumer already uses, so neither the Grafana
+# Deployment nor mcp-grafana changes.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: observability-secrets
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: vault-observability
+  target:
+    name: observability-secrets
+    creationPolicy: Owner
+  dataFrom:
+    # Whole-secret pull rather than key-by-key: adding a key in Vault then
+    # reaches the cluster without editing this file.
+    - extract:
+        key: observability/grafana

k8s/secrets/kustomization.yaml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - clustersecretstore.yaml
+  - externalsecret.yaml