Every hop was starting its own trace: a single end-to-end transaction through
LibreChat, Kong and the backends appeared in Tempo as a dozen unrelated traces,
one per service. Verified on twelve consecutive traces, each containing exactly
one service.
Kong receives a W3C traceparent -- it is visible in the http-log records -- but
without a propagation block it neither continues the caller's trace nor passes
one downstream, so it minted a fresh trace id per request and the backends did
the same.
extract lists w3c first with b3 and jaeger as fallbacks, since the callers here
are a mix of OTel SDKs; inject is w3c only, which is what the Python and Java
agents downstream expect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Point OTEL_TRACES_ENDPOINT at Tempo instead of Jaeger. The variable is a
cluster-wide default, so this moves all five domains at once; Jaeger stays
deployed but no longer receives anything. Tempo derives RED span metrics
and the service graph from these traces on its own, and its trace ids
correlate with the audit logs below.
Add a Loki output to the audit collector. Kong's http-log payload is a
JSON object per request and Loki has no endpoint that accepts it, so
Fluent Bit is the translator between the two formats.
The `domain` label comes from $TAG, which the http input derives from the
request URI. That makes the path a domain posts to its identity here, so
one output block covers every domain and onboarding the next one is a
single AUDIT_LOG_ENDPOINT change on its side rather than another block.
The file output stays on Match * so the on-disk audit trail continues for
every domain, which keeps the switch reversible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Receives the http-log plugin's POSTs and dumps each record as a JSON
line to a PVC-backed file. Applied directly (kubectl), same as Jaeger
in this namespace — not Flux-managed yet.
cert-manager-issued Certificates never synced to Konnect as KongCertificate
objects through the Gateway API listener path, so dataplanes fell back to
Kong's default self-signed cert. Reuse the working kong-namespace pattern
(a static wildcard cert) via ExternalSecret instead.
Stéphane Tailland