Two ceilings had to clear 30 KB prompts, and neither alone was enough:
nginx's client_body_buffer_size, which decides whether the body stays in
memory at all, and ai-proxy's max_request_body_size, which decides whether
the plugin will introspect it. With the first at 8k the body went to a
temp file and kong.request.get_raw_body() returned nil, so log_payloads
and log_statistics were enabled and empty, and no kong_ai_llm_* metric
ever appeared.
256k here against the 8k the domain gateways keep. Affordable because this
gateway is low-traffic; the buffer is per connection carrying a body, so
copying the value to the domain gateways needs that arithmetic redone for
their volume.
Storage cost measured rather than assumed: 18 LLM requests a day at
roughly 60 KB of prompt and completion is 1.1 MiB/day, against 425 MiB/day
of Kong audit records already flowing. My earlier warning that the audit
trail would grow fast was wrong at this volume.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stéphane Tailland
a26e57b2c5