A tool-calling agent re-sends the whole tool result on every iteration of its
loop, so a single question against a tool returning a full collection burns the
old budget in three turns and answers 429 in the middle of reasoning. Pagination
on those tools is the real fix; this buys room until then and stays well inside
the 500k global ceiling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The AI gateway's openid-connect plugin still carried the cluster-wide
dataplane-template-defaults issuer, https://keycloak.sttlab.eu/realms/sttlab.
Every other domain gateway overrides it to the lab realm; this one was missed.
The plugin therefore fetched JWKS from a realm that never signed the token and
rejected it with "invalid signature", which reaches the caller as a bare 401
"Authentication required". Combined with the missing llm-proxy audience in the
realm, the OIDC path on this gateway had never authenticated anyone — only the
key-auth path LibreChat uses was ever exercised.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two ceilings had to clear 30 KB prompts, and neither alone was enough:
nginx's client_body_buffer_size, which decides whether the body stays in
memory at all, and ai-proxy's max_request_body_size, which decides whether
the plugin will introspect it. With the first at 8k the body went to a
temp file and kong.request.get_raw_body() returned nil, so log_payloads
and log_statistics were enabled and empty, and no kong_ai_llm_* metric
ever appeared.
256k here against the 8k the domain gateways keep. Affordable because this
gateway is low-traffic; the buffer is per connection carrying a body, so
copying the value to the domain gateways needs that arithmetic redone for
their volume.
Storage cost measured rather than assumed: 18 LLM requests a day at
roughly 60 KB of prompt and completion is 1.1 MiB/day, against 425 MiB/day
of Kong audit records already flowing. My earlier warning that the audit
trail would grow fast was wrong at this volume.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dependsOn still named flux-system, where it used to live. Flux reports
"dependency not found" and holds this Kustomization not ready rather than
failing loudly, so the Kong configuration silently stopped reconciling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ai-gateway-kustomization sat in flux-system, so the team's own bootstrap
was writing into a namespace the team has no access to — the thing the
move was meant to end. It follows the others into llm, with an explicit
sourceRef namespace since k8s-platform stays in flux-system.
The control-plane ExternalSecret goes with it, for the same reason it did
in the dataplane repos: it only lived in flux-system because
substituteFrom resolves in the Kustomization's own namespace.
And this retires a workaround. proxy-kustomization inlined
OTEL_TRACES_ENDPOINT because it must stay in llm — it substitutes from a
Secret holding OPENROUTER_BEARER and LIBRECHAT_APIKEY, which must not be
copied into flux-system — and could therefore not read
dataplane-template-defaults, which existed only there. k8s-platform now
places that ConfigMap in every domain namespace, so the reference works
and the duplicated value goes. Changing the cluster trace backend is one
edit again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Twenty-four Flux objects across eleven repos, none of them managed by
Flux. Each repo declares its own GitRepository and Kustomization, which
is the right ownership model, but nothing ever watched the directory
those files live in — so editing one in git changed nothing until
someone remembered to kubectl apply it. That cost real time twice this
week: the CONTROL_PLANE_REF_NAME rollout looked applied and was not, and
LLM_INPUT_COST failed the build with "variable not set" while sitting
correctly in git.
The gap is the bootstrap Kustomization that `flux bootstrap` would have
created and flux-operator does not. Add one per repo, watching that
repo's own ./dev, living in ./dev so it manages itself. Nothing moves
between repos; each domain keeps its entrypoints.
This is the first of eleven, starting with platform-llm because its cost
settings are blocked behind exactly this.
Needs one kubectl apply to exist. That is the last one this repo needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four settings, all of them off, which is why the AI gateway produced
HTTP-level telemetry and nothing about the model.
logging.log_payloads the prompt and the completion, in the log
records and — since OTel instrumentation is on
here — in the spans too
logging.log_statistics model usage and token counts
prometheus.ai_metrics kong_ai_llm_* : requests, tokens by type, cost,
provider latency, per model and per consumer
input_cost / output_cost cost per 1M tokens, which is what makes
kong_ai_llm_cost_total mean anything
Also removes this morning's pre-function/post-function body capture from
this gateway. It was the wrong tool for the LLM route: Kong's own
implementation reassembles a streamed SSE answer, while accumulating raw
chunks in kong.ctx.shared does not. The five business domains keep theirs
— their MCP traffic is not streamed the same way.
The costs are a snapshot, checked against OpenRouter's price list for
google/gemini-3.7-flash on 2026-08-27: $0.375 in, $1.875 out per 1M
tokens. They are substitution variables rather than literals precisely
because a stale figure produces confidently wrong money rather than an
error.
Note what this does NOT fix: usage is missing on streamed responses
because the OpenAI-compatible API only reports it when the caller asks
for stream_options.include_usage, and that is LibreChat's request to
make, not Kong's. Expect token counts to stay partial until then.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The audit trail recorded metadata only for this gateway, so the prompts
sent to the LLM and the completions coming back — the one thing worth
auditing on an AI gateway — were never written down. The five business
domains have captured bodies since they were built; this gateway declares
its own copy of the observability plugins and never got them.
Adds the same pre-function/post-function pair the domains use, bound to
the proxy service, and the custom_fields_by_lua that reads them back in
the log phase. If the capture plugins are ever removed the fields go nil
and are silently omitted, so the log degrades to metadata rather than
breaking.
The response accumulator earns its keep here more than elsewhere: LLM
answers stream as SSE, so body_filter runs once per chunk and the
complete answer only exists after the last one.
Still dev-only, for the same reason as the rest of this file: it puts
prompt and completion content in the log stream.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This gateway declared its own copy of the shared observability plugins,
claiming parity with k8s-platform's dataplane-globalpolicy-template. The
propagation block was later added to that template and never mirrored
here, so while the five business domains continued the caller's trace,
this one minted a fresh trace id on every request and injected nothing
upstream.
That breaks end-to-end tracing at the worst possible hop: LibreChat
enters here, so a transaction that fans out to MCP, the LLM and the
backend was recorded as a set of unrelated traces. Verified live before
the fix — the domain gateways reported
extract: [w3c, b3, jaeger] / inject: [w3c], this one reported
extract: null / inject: null.
Copy the block across and flag in the header comment that the parity is
manual and has already drifted once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This Kustomization pulled dataplane-template-defaults via substituteFrom
with optional: false, but that ConfigMap is deliberately single-instance
in flux-system and this object lives in the llm namespace, so the build
failed outright: configmaps "dataplane-template-defaults" not found.
That is why the stack was never in the cluster and the llm KongService,
KongRoute and plugin bindings had been applied by hand instead.
Moving the Kustomization to flux-system would fix the lookup and break
something worse: its substitution Secret holds OPENROUTER_BEARER and
LIBRECHAT_APIKEY, and copying real credentials into flux-system is
exactly what the Konnect PAT comment in gateway-kustomization.yaml
refuses to do.
OTEL_TRACES_ENDPOINT was the only key this stack took from that
ConfigMap, so inline it and drop the reference. The trade-off is
recorded next to the value: changing the cluster trace backend now takes
an edit here too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two hardcoded values tied this repo to one specific Konnect org, and
replacing that org broke both.
CONTROL_PLANE_ID was pasted into gateway-kustomization.yaml. It now
comes from Vault through External Secrets, into a Secret in flux-system
that postBuild.substituteFrom reads. flux-system is acceptable for the
id — it is visible in the Konnect UI URL — but not for the token, which
keeps going straight to the domain namespace as before.
CONTROL_PLANE_REF_NAME held the random suffix the Gateway operator gives
its auto-created mirror CR, looked up by hand. The new org recreated
those CRs under new suffixes, so the ref named an object that no longer
existed and the domain's Kong entities sat Programmed=False. It now
names the ${NAME}-cp mirror declared in k8s-platform's
dataplane-template — derived from NAME, stable across Gateway
recreations and org changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This gateway had none of the three: no prometheus, opentelemetry or http-log
plugin, so it produced only Kong's own node and shared-dict metrics and was
invisible in Tempo and Loki.
The five business domains inherit these from k8s-platform's
dataplane-globalpolicy-template, but that template also carries
openid-connect, rate-limiting and cors, and this gateway already defines its
own AI-specific versions of all three. Applying it here would duplicate them,
so the observability plugins are declared locally with the same configuration
as the shared template.
prometheus and opentelemetry bind control-plane-wide since they describe the
gateway itself; the audit log binds to the proxy service, matching how the
domains bind theirs.
The trace endpoint now comes from the shared dataplane-template-defaults
ConfigMap rather than being repeated here, so a change of trace backend does
not need a second edit. Its only key overlapping the inline block is
OIDC_ISSUER, and both carry the same value.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland