Stéphane Tailland
a26e57b2c5
Two ceilings had to clear 30 KB prompts, and neither alone was enough: nginx's client_body_buffer_size, which decides whether the body stays in memory at all, and ai-proxy's max_request_body_size, which decides whether the plugin will introspect it. With the first at 8k the body went to a temp file and kong.request.get_raw_body() returned nil, so log_payloads and log_statistics were enabled and empty, and no kong_ai_llm_* metric ever appeared. 256k here against the 8k the domain gateways keep. Affordable because this gateway is low-traffic; the buffer is per connection carrying a body, so copying the value to the domain gateways needs that arithmetic redone for their volume. Storage cost measured rather than assumed: 18 LLM requests a day at roughly 60 KB of prompt and completion is 1.1 MiB/day, against 425 MiB/day of Kong audit records already flowing. My earlier warning that the audit trail would grow fast was wrong at this volume. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
42 lines
1.8 KiB
YAML
42 lines
1.8 KiB
YAML
apiVersion: configuration.konghq.com/v1
|
|
kind: KongPlugin
|
|
metadata:
|
|
name: ${NAME}-llm-ai-proxy-advanced
|
|
namespace: ${NAMESPACE}
|
|
plugin: ai-proxy-advanced
|
|
config:
|
|
# The plugin's own ceiling on what it will introspect, distinct from nginx's
|
|
# client_body_buffer_size — both have to clear 30 KB prompts or the payload
|
|
# and usage logging stay empty. Raised together; neither alone is enough.
|
|
max_request_body_size: 262144
|
|
targets:
|
|
- route_type: llm/v1/chat
|
|
auth:
|
|
header_name: Authorization
|
|
header_value: ${OPENROUTER_BEARER}
|
|
# dev only, like the rest of this gateway's audit trail: log_payloads
|
|
# puts the prompt and the completion in the log stream, and -- because
|
|
# OpenTelemetry instrumentation is on here -- in the spans as well.
|
|
#
|
|
# This is what the pre-function/post-function capture in
|
|
# observability.yaml stood in for on this route. Kong's own
|
|
# implementation reassembles a streamed SSE answer; accumulating raw
|
|
# chunks does not.
|
|
logging:
|
|
log_payloads: true
|
|
# Model usage and token counts in the log records. With this off Kong
|
|
# counted nothing of its own, which is half of why usage was missing.
|
|
log_statistics: true
|
|
model:
|
|
provider: openai
|
|
# name deliberately unset: the client's own `model` field is passed
|
|
# through to OpenRouter.
|
|
options:
|
|
upstream_url: ${OPENROUTER_UPSTREAM_URL}
|
|
# Cost per 1M tokens, in the provider's currency. Kong multiplies
|
|
# these by the token counts to produce kong_ai_llm_cost_total, so a
|
|
# stale figure yields confidently wrong money rather than an error.
|
|
# Re-check them whenever the model or the contract changes.
|
|
input_cost: ${LLM_INPUT_COST}
|
|
output_cost: ${LLM_OUTPUT_COST}
|