This gateway declared its own copy of the shared observability plugins,
claiming parity with k8s-platform's dataplane-globalpolicy-template. The
propagation block was later added to that template and never mirrored
here, so while the five business domains continued the caller's trace,
this one minted a fresh trace id on every request and injected nothing
upstream.
That breaks end-to-end tracing at the worst possible hop: LibreChat
enters here, so a transaction that fans out to MCP, the LLM and the
backend was recorded as a set of unrelated traces. Verified live before
the fix — the domain gateways reported
extract: [w3c, b3, jaeger] / inject: [w3c], this one reported
extract: null / inject: null.
Copy the block across and flag in the header comment that the parity is
manual and has already drifted once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This Kustomization pulled dataplane-template-defaults via substituteFrom
with optional: false, but that ConfigMap is deliberately single-instance
in flux-system and this object lives in the llm namespace, so the build
failed outright: configmaps "dataplane-template-defaults" not found.
That is why the stack was never in the cluster and the llm KongService,
KongRoute and plugin bindings had been applied by hand instead.
Moving the Kustomization to flux-system would fix the lookup and break
something worse: its substitution Secret holds OPENROUTER_BEARER and
LIBRECHAT_APIKEY, and copying real credentials into flux-system is
exactly what the Konnect PAT comment in gateway-kustomization.yaml
refuses to do.
OTEL_TRACES_ENDPOINT was the only key this stack took from that
ConfigMap, so inline it and drop the reference. The trade-off is
recorded next to the value: changing the cluster trace backend now takes
an edit here too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two hardcoded values tied this repo to one specific Konnect org, and
replacing that org broke both.
CONTROL_PLANE_ID was pasted into gateway-kustomization.yaml. It now
comes from Vault through External Secrets, into a Secret in flux-system
that postBuild.substituteFrom reads. flux-system is acceptable for the
id — it is visible in the Konnect UI URL — but not for the token, which
keeps going straight to the domain namespace as before.
CONTROL_PLANE_REF_NAME held the random suffix the Gateway operator gives
its auto-created mirror CR, looked up by hand. The new org recreated
those CRs under new suffixes, so the ref named an object that no longer
existed and the domain's Kong entities sat Programmed=False. It now
names the ${NAME}-cp mirror declared in k8s-platform's
dataplane-template — derived from NAME, stable across Gateway
recreations and org changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This gateway had none of the three: no prometheus, opentelemetry or http-log
plugin, so it produced only Kong's own node and shared-dict metrics and was
invisible in Tempo and Loki.
The five business domains inherit these from k8s-platform's
dataplane-globalpolicy-template, but that template also carries
openid-connect, rate-limiting and cors, and this gateway already defines its
own AI-specific versions of all three. Applying it here would duplicate them,
so the observability plugins are declared locally with the same configuration
as the shared template.
prometheus and opentelemetry bind control-plane-wide since they describe the
gateway itself; the audit log binds to the proxy service, matching how the
domains bind theirs.
The trace endpoint now comes from the shared dataplane-template-defaults
ConfigMap rather than being repeated here, so a change of trace backend does
not need a second edit. Its only key overlapping the inline block is
OIDC_ISSUER, and both carry the same value.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland