Stéphane Tailland
a26e57b2c5
Two ceilings had to clear 30 KB prompts, and neither alone was enough: nginx's client_body_buffer_size, which decides whether the body stays in memory at all, and ai-proxy's max_request_body_size, which decides whether the plugin will introspect it. With the first at 8k the body went to a temp file and kong.request.get_raw_body() returned nil, so log_payloads and log_statistics were enabled and empty, and no kong_ai_llm_* metric ever appeared. 256k here against the 8k the domain gateways keep. Affordable because this gateway is low-traffic; the buffer is per connection carrying a body, so copying the value to the domain gateways needs that arithmetic redone for their volume. Storage cost measured rather than assumed: 18 LLM requests a day at roughly 60 KB of prompt and completion is 1.1 MiB/day, against 425 MiB/day of Kong audit records already flowing. My earlier warning that the audit trail would grow fast was wrong at this volume. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
platform-llm
Platform-owned llm bounded context — Gateway infrastructure and the
shared LLM proxy in one repo. Never split per functional domain, and never
split between infra/routing here either: both are owned by the same
(platform) team, unlike catalog's dataplane-catalog/catalog-api split
(feature team vs. platform team, a real access boundary that doesn't exist
for this domain).
What it does
Fronts OpenRouter (https://openrouter.ai) via one proxy route
(/llm/v1/chat/completions) — the client's own model field selects among
any OpenRouter-hosted model, not just a fixed set Kong picks.
Active policies:
ai-rate-limiting-advanced, token-based, both global (across all callers) and per-client (per authenticated consumer).ai-prompt-guard(regex) andai-semantic-prompt-guard(embedding-based, backed by Redis Stack) for prompt-injection/jailbreak safeguarding.- Two-tier auth:
key-auth(static, shared key) for chatbot UIs that can't do OIDC on their outbound LLM connection (LibreChat and similar tools — structural limitation of that whole category, not fixable client-side),openid-connectfor agents built in-house. Either succeeds; a request passing neither is rejected (request-terminationon the fallback anonymous consumer).
Not yet active: PII redaction (ai-sanitizer) — needs a private Kong
Cloudsmith registry image and an AI Gateway Enterprise entitlement, blocked
pending that access. Everything else in this repo is unaffected.
Structure
dev/ai-gateway-kustomization.yaml+dev/ai-gateway-externalsecret.yaml— Gateway/DataPlane/Control Plane infra (same sharedk8s-platformtemplate every domain uses).dev/proxy-kustomization.yaml+dev/proxy-externalsecret.yaml— the LLM proxy itself (./kong).dev/gitrepository.yaml— this repo, self-referential.