The domains' tools: arrays are hand-maintained, and that has now cost two
defects an agent found the hard way — create-product could never succeed, and
list-products cannot paginate, which let a model answer a product count that
looked right and did not come from the data.
Five limits measured against decK v1.64.0 and Kong Gateway 3.15, each
reproduced rather than read: per-tool ACLs cannot express a consumer-group
model and fail silently; the four MCP behavioural annotations are not
generated; response schemas are not generated although the plugin consumes them
for structuredOutput; servers is taken as the upstream target; every content
type is copied and every operation exposed.
FastMCP run on the same spec is recorded as a comparison point — it closes
three of the five out of the box — with the reason we are not adopting it: it
would move per-tool authorization out of the realm and into application code.
Also records that MCP Composer and MCP Runner never shipped, so none of this is
waiting on an imminent release.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ai-proxy's log_payloads and log_statistics produced nothing on the AI
gateway despite being enabled. The cause was not the plugin's own
max_request_body_size but nginx's client_body_buffer_size, left at its 8k
default: past it the body goes to a temp file, and
kong.request.get_raw_body() returns nil for a body on disk, so any plugin
reading the body silently stops working. Prompts on that gateway run to 30
KB. Measured, not guessed — request.size was 30369 on the last call.
Expose it as ${CLIENT_BODY_BUFFER_SIZE} with the 8k default every domain
keeps, rather than raising it globally. The buffer is allocated per
connection carrying a body, so a value that is free on a gateway seeing 18
requests a day is not free on one serving real traffic.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
postBuild.substituteFrom can only read a ConfigMap in its own namespace.
dataplane-template-defaults existed only in flux-system, so every gateway
Kustomization had to live in flux-system to reach it — a namespace the
feature teams have no access to. Their own gateway configuration ended up
outside their reach as a side effect of a substitution rule, not as a
governance decision.
Add a namespace-less copy of the ConfigMap under
kong-platform/domain-defaults, plus one Kustomization per domain in
flux/domain-defaults.yaml differing only by targetNamespace. One source
file, one edit when a default changes, six copies that follow.
Platform-owned deliberately: these are the defaults a team inherits, not
ones it sets, so they stay in flux-system while everything a team does own
moves out of it.
This is the precondition, not the move itself. The flux-system copy stays
until the gateway Kustomizations relocate, and they cannot be patched into
a new namespace — that is a delete and recreate, with a window where the
gateway is unreconciled.
It also retires a workaround: platform-llm inlined OTEL_TRACES_ENDPOINT
because its Kustomization had to sit in the llm namespace, holding real
credentials it must not put in flux-system, and so could not read this
ConfigMap. With the ConfigMap in llm, that inlining can go.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Kong entity names its control plane through
controlPlaneRef.konnectNamespacedRef (this operator version exposes no
konnectID ref type), and the only mirror CR available was the one the
Gateway operator auto-creates with a random suffix. That suffix was
looked up by hand and pasted into CONTROL_PLANE_REF_NAME in 11 places
across 11 repos, where it went stale silently whenever the Gateway was
recreated. Replacing the Konnect org did exactly that: all 11 refs named
CRs that no longer existed and every service, route and plugin binding
sat Programmed=False.
Declare a mirror CR named ${NAME}-cp instead, so the refs are derived
from NAME and never need touching again. source: Mirror adopts an
existing control plane rather than owning it — deleting the CR leaves
the control plane in Konnect intact (verified: count unchanged across a
delete), so it coexists with the operator's own mirror of the same id.
The id itself comes from Vault via External Secrets, so a new Konnect
org now needs no edit in any repo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every hop was starting its own trace: a single end-to-end transaction through
LibreChat, Kong and the backends appeared in Tempo as a dozen unrelated traces,
one per service. Verified on twelve consecutive traces, each containing exactly
one service.
Kong receives a W3C traceparent -- it is visible in the http-log records -- but
without a propagation block it neither continues the caller's trace nor passes
one downstream, so it minted a fresh trace id per request and the backends did
the same.
extract lists w3c first with b3 and jaeger as fallbacks, since the callers here
are a mix of OTel SDKs; inject is w3c only, which is what the Python and Java
agents downstream expect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Point OTEL_TRACES_ENDPOINT at Tempo instead of Jaeger. The variable is a
cluster-wide default, so this moves all five domains at once; Jaeger stays
deployed but no longer receives anything. Tempo derives RED span metrics
and the service graph from these traces on its own, and its trace ids
correlate with the audit logs below.
Add a Loki output to the audit collector. Kong's http-log payload is a
JSON object per request and Loki has no endpoint that accepts it, so
Fluent Bit is the translator between the two formats.
The `domain` label comes from $TAG, which the http input derives from the
request URI. That makes the path a domain posts to its identity here, so
one output block covers every domain and onboarding the next one is a
single AUDIT_LOG_ENDPOINT change on its side rather than another block.
The file output stays on Match * so the on-disk audit trail continues for
every domain, which keeps the switch reversible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Receives the http-log plugin's POSTs and dumps each record as a JSON
line to a PVC-backed file. Applied directly (kubectl), same as Jaeger
in this namespace — not Flux-managed yet.
cert-manager-issued Certificates never synced to Konnect as KongCertificate
objects through the Gateway API listener path, so dataplanes fell back to
Kong's default self-signed cert. Reuse the working kong-namespace pattern
(a static wildcard cert) via ExternalSecret instead.
Stéphane Tailland