configmap-realm.yaml carried fourteen client secrets and five user
passwords in clear, in a tracked file. Five were there from its first
commit; I added the fourteenth yesterday by following the existing pattern
without questioning it. Making this file authoritative on every start did
not increase the exposure, but it did make the file the place those
secrets live for good — and it is inconsistent with the rest of the
estate, where infra-observability and kong-v2 both take theirs from Vault.
Keycloak resolves ${PLACEHOLDER} in a realm import file from the
environment, so the structure stays in git and the values move to Vault at
secrets/keycloak/classic-models-realm, reaching the import container
through External Secrets.
A store of its own, vault-keycloak, rather than reusing vault-kong-v2 or
vault-observability: each carries a policy scoped to its own path, so a
reader for one estate cannot see another's. Widening an existing policy
would have handed Kong's reader every client secret in the realm.
dataFrom.extract pulls the whole path, so adding a client means adding its
secret in Vault with no edit to the ExternalSecret.
Verified after the switch: the import logs "Realm 'classic-models' imported"
under OVERWRITE_EXISTING, and all three credential paths still work — the
broker's client_credentials, librechat with pm-user's password, and the
Catalogue agent's exchange reaching catalog-mcp with 200. Re-checked that
none of the nineteen original values appears anywhere in the file.
Note the values themselves were committed and are in this repo's history.
They are lab credentials behind a private CA with no real data, but they
should be rotated if that ever stops being true.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
configmap-realm.yaml was a first-install seed: `start --import-realm`
skips a realm that already exists and logs "Import skipped", so editing
the file changed nothing on a restart and the truth drifted into the
database. It already had — four scope grants existed only in the running
realm.
There is no option to make the startup import overwrite. The --override
flag exists only on the standalone `import` command, and the Keycloak
documentation is explicit that re-creating realms means running that
command before starting the server. So the import moves into an init
container that runs `import --dir ... --override true`, and --import-realm
comes off the main container's args.
The init container empties JAVA_OPTS_APPEND for itself. It inherits the
main container's env, which loads the OpenTelemetry agent from a volume
the other init container downloads into and this one does not mount;
inherited verbatim, the JVM fails before the import runs.
What this costs, deliberately: the realm is replaced at every start, so
anything created at runtime and not written back to this file is lost, and
active sessions are dropped. Checked before enabling it — the five users
in the realm are the five declared here, and the only live sessions were
two LibreChat ones. Nothing is to be created through the admin console
expecting it to last.
Verified after the switch: the import logs "Strategy: OVERWRITE_EXISTING",
the realm still has its 21 clients with no optionalClientScopes
divergence, and the full identity chain still returns 200 — pm-user token,
exchanged by the Catalogue agent, accepted by catalog-mcp.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Point the OTel Java agent at Tempo instead of Jaeger, so Keycloak's traces
land in the same backend as the rest of the stack and correlate with the
Loki logs of the same requests.
Annotate the pod for scraping. KC_METRICS_ENABLED already served /metrics
on the management port, so this only makes Prometheus discover it — on
port 9000, never on the application port.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland