Stéphane Tailland
a26e57b2c5
Two ceilings had to clear 30 KB prompts, and neither alone was enough: nginx's client_body_buffer_size, which decides whether the body stays in memory at all, and ai-proxy's max_request_body_size, which decides whether the plugin will introspect it. With the first at 8k the body went to a temp file and kong.request.get_raw_body() returned nil, so log_payloads and log_statistics were enabled and empty, and no kong_ai_llm_* metric ever appeared. 256k here against the 8k the domain gateways keep. Affordable because this gateway is low-traffic; the buffer is per connection carrying a body, so copying the value to the domain gateways needs that arithmetic redone for their volume. Storage cost measured rather than assumed: 18 LLM requests a day at roughly 60 KB of prompt and completion is 1.1 MiB/day, against 425 MiB/day of Kong audit records already flowing. My earlier warning that the audit trail would grow fast was wrong at this volume. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
41 lines
1.6 KiB
YAML
41 lines
1.6 KiB
YAML
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: llm-dev-ai-gateway
|
|
namespace: llm
|
|
spec:
|
|
interval: 10m
|
|
sourceRef:
|
|
kind: GitRepository
|
|
name: k8s-platform
|
|
# k8s-platform lives in flux-system: the templates are the platform's,
|
|
# this Kustomization is the team's. sourceRef resolves in the
|
|
# Kustomization's own namespace unless told otherwise.
|
|
namespace: flux-system
|
|
path: ./dataplane-template
|
|
prune: true
|
|
postBuild:
|
|
substitute:
|
|
NAME: llm-dev
|
|
NAMESPACE: llm
|
|
# 256k, against the 8k the other domains keep. Prompts here carry the
|
|
# conversation and the MCP tool definitions and already run to 30 KB;
|
|
# below this nginx spills the body to a temp file and
|
|
# kong.request.get_raw_body() returns nil, which is what left ai-proxy's
|
|
# log_payloads and log_statistics producing nothing at all.
|
|
#
|
|
# Affordable precisely because this gateway is low-traffic — the buffer is
|
|
# per connection carrying a body. Do not copy it to the domain gateways
|
|
# without doing that arithmetic for their volume.
|
|
CLIENT_BODY_BUFFER_SIZE: 256k
|
|
HOSTNAME: llm-dev.sttlab.local
|
|
substituteFrom:
|
|
- kind: ConfigMap
|
|
name: dataplane-template-defaults
|
|
# CONTROL_PLANE_ID comes from Vault via ESO instead of being hardcoded
|
|
# here: a new Konnect org then needs no edit in this repo. Not a secret —
|
|
# the id is visible in the Konnect UI URL — so flux-system is fine for it,
|
|
# unlike the token.
|
|
- kind: Secret
|
|
name: llm-dev-control-plane
|