ai-proxy's log_payloads and log_statistics produced nothing on the AI
gateway despite being enabled. The cause was not the plugin's own
max_request_body_size but nginx's client_body_buffer_size, left at its 8k
default: past it the body goes to a temp file, and
kong.request.get_raw_body() returns nil for a body on disk, so any plugin
reading the body silently stops working. Prompts on that gateway run to 30
KB. Measured, not guessed — request.size was 30369 on the last call.
Expose it as ${CLIENT_BODY_BUFFER_SIZE} with the 8k default every domain
keeps, rather than raising it globally. The buffer is allocated per
connection carrying a body, so a value that is free on a gateway seeing 18
requests a day is not free on one serving real traffic.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
postBuild.substituteFrom can only read a ConfigMap in its own namespace.
dataplane-template-defaults existed only in flux-system, so every gateway
Kustomization had to live in flux-system to reach it — a namespace the
feature teams have no access to. Their own gateway configuration ended up
outside their reach as a side effect of a substitution rule, not as a
governance decision.
Add a namespace-less copy of the ConfigMap under
kong-platform/domain-defaults, plus one Kustomization per domain in
flux/domain-defaults.yaml differing only by targetNamespace. One source
file, one edit when a default changes, six copies that follow.
Platform-owned deliberately: these are the defaults a team inherits, not
ones it sets, so they stay in flux-system while everything a team does own
moves out of it.
This is the precondition, not the move itself. The flux-system copy stays
until the gateway Kustomizations relocate, and they cannot be patched into
a new namespace — that is a delete and recreate, with a window where the
gateway is unreconciled.
It also retires a workaround: platform-llm inlined OTEL_TRACES_ENDPOINT
because its Kustomization had to sit in the llm namespace, holding real
credentials it must not put in flux-system, and so could not read this
ConfigMap. With the ConfigMap in llm, that inlining can go.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Point OTEL_TRACES_ENDPOINT at Tempo instead of Jaeger. The variable is a
cluster-wide default, so this moves all five domains at once; Jaeger stays
deployed but no longer receives anything. Tempo derives RED span metrics
and the service graph from these traces on its own, and its trace ids
correlate with the audit logs below.
Add a Loki output to the audit collector. Kong's http-log payload is a
JSON object per request and Loki has no endpoint that accepts it, so
Fluent Bit is the translator between the two formats.
The `domain` label comes from $TAG, which the http input derives from the
request URI. That makes the path a domain posts to its identity here, so
one output block covers every domain and onboarding the next one is a
single AUDIT_LOG_ENDPOINT change on its side rather than another block.
The file output stays on Match * so the on-disk audit trail continues for
every domain, which keeps the switch reversible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cert-manager-issued Certificates never synced to Konnect as KongCertificate
objects through the Gateway API listener path, so dataplanes fell back to
Kong's default self-signed cert. Reuse the working kong-namespace pattern
(a static wildcard cert) via ExternalSecret instead.
Stéphane Tailland