Two ceilings had to clear 30 KB prompts, and neither alone was enough:
nginx's client_body_buffer_size, which decides whether the body stays in
memory at all, and ai-proxy's max_request_body_size, which decides whether
the plugin will introspect it. With the first at 8k the body went to a
temp file and kong.request.get_raw_body() returned nil, so log_payloads
and log_statistics were enabled and empty, and no kong_ai_llm_* metric
ever appeared.
256k here against the 8k the domain gateways keep. Affordable because this
gateway is low-traffic; the buffer is per connection carrying a body, so
copying the value to the domain gateways needs that arithmetic redone for
their volume.
Storage cost measured rather than assumed: 18 LLM requests a day at
roughly 60 KB of prompt and completion is 1.1 MiB/day, against 425 MiB/day
of Kong audit records already flowing. My earlier warning that the audit
trail would grow fast was wrong at this volume.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four settings, all of them off, which is why the AI gateway produced
HTTP-level telemetry and nothing about the model.
logging.log_payloads the prompt and the completion, in the log
records and — since OTel instrumentation is on
here — in the spans too
logging.log_statistics model usage and token counts
prometheus.ai_metrics kong_ai_llm_* : requests, tokens by type, cost,
provider latency, per model and per consumer
input_cost / output_cost cost per 1M tokens, which is what makes
kong_ai_llm_cost_total mean anything
Also removes this morning's pre-function/post-function body capture from
this gateway. It was the wrong tool for the LLM route: Kong's own
implementation reassembles a streamed SSE answer, while accumulating raw
chunks in kong.ctx.shared does not. The five business domains keep theirs
— their MCP traffic is not streamed the same way.
The costs are a snapshot, checked against OpenRouter's price list for
google/gemini-3.7-flash on 2026-08-27: $0.375 in, $1.875 out per 1M
tokens. They are substitution variables rather than literals precisely
because a stale figure produces confidently wrong money rather than an
error.
Note what this does NOT fix: usage is missing on streamed responses
because the OpenAI-compatible API only reports it when the caller asks
for stream_options.include_usage, and that is LibreChat's request to
make, not Kong's. Expect token counts to stay partial until then.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland