Every Kong entity names its control plane through
controlPlaneRef.konnectNamespacedRef (this operator version exposes no
konnectID ref type), and the only mirror CR available was the one the
Gateway operator auto-creates with a random suffix. That suffix was
looked up by hand and pasted into CONTROL_PLANE_REF_NAME in 11 places
across 11 repos, where it went stale silently whenever the Gateway was
recreated. Replacing the Konnect org did exactly that: all 11 refs named
CRs that no longer existed and every service, route and plugin binding
sat Programmed=False.
Declare a mirror CR named ${NAME}-cp instead, so the refs are derived
from NAME and never need touching again. source: Mirror adopts an
existing control plane rather than owning it — deleting the CR leaves
the control plane in Konnect intact (verified: count unchanged across a
delete), so it coexists with the operator's own mirror of the same id.
The id itself comes from Vault via External Secrets, so a new Konnect
org now needs no edit in any repo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every hop was starting its own trace: a single end-to-end transaction through
LibreChat, Kong and the backends appeared in Tempo as a dozen unrelated traces,
one per service. Verified on twelve consecutive traces, each containing exactly
one service.
Kong receives a W3C traceparent -- it is visible in the http-log records -- but
without a propagation block it neither continues the caller's trace nor passes
one downstream, so it minted a fresh trace id per request and the backends did
the same.
extract lists w3c first with b3 and jaeger as fallbacks, since the callers here
are a mix of OTel SDKs; inject is w3c only, which is what the Python and Java
agents downstream expect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Point OTEL_TRACES_ENDPOINT at Tempo instead of Jaeger. The variable is a
cluster-wide default, so this moves all five domains at once; Jaeger stays
deployed but no longer receives anything. Tempo derives RED span metrics
and the service graph from these traces on its own, and its trace ids
correlate with the audit logs below.
Add a Loki output to the audit collector. Kong's http-log payload is a
JSON object per request and Loki has no endpoint that accepts it, so
Fluent Bit is the translator between the two formats.
The `domain` label comes from $TAG, which the http input derives from the
request URI. That makes the path a domain posts to its identity here, so
one output block covers every domain and onboarding the next one is a
single AUDIT_LOG_ENDPOINT change on its side rather than another block.
The file output stays on Match * so the on-disk audit trail continues for
every domain, which keeps the switch reversible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Receives the http-log plugin's POSTs and dumps each record as a JSON
line to a PVC-backed file. Applied directly (kubectl), same as Jaeger
in this namespace — not Flux-managed yet.
cert-manager-issued Certificates never synced to Konnect as KongCertificate
objects through the Gateway API listener path, so dataplanes fell back to
Kong's default self-signed cert. Reuse the working kong-namespace pattern
(a static wildcard cert) via ExternalSecret instead.
Stéphane Tailland