Stéphane Tailland
40 lines
1.8 KiB
Markdown
40 lines
1.8 KiB
Markdown
# platform-llm
|
|
|
|
Platform-owned `llm` bounded context — Gateway infrastructure and the
|
|
shared LLM proxy in one repo. Never split per functional domain, and never
|
|
split between infra/routing here either: both are owned by the same
|
|
(platform) team, unlike `catalog`'s `dataplane-catalog`/`catalog-api` split
|
|
(feature team vs. platform team, a real access boundary that doesn't exist
|
|
for this domain).
|
|
|
|
## What it does
|
|
|
|
Fronts OpenRouter (`https://openrouter.ai`) via one proxy route
|
|
(`/llm/v1/chat/completions`) — the client's own `model` field selects among
|
|
any OpenRouter-hosted model, not just a fixed set Kong picks.
|
|
|
|
Active policies:
|
|
- `ai-rate-limiting-advanced`, token-based, both global (across all
|
|
callers) and per-client (per authenticated consumer).
|
|
- `ai-prompt-guard` (regex) and `ai-semantic-prompt-guard` (embedding-based,
|
|
backed by Redis Stack) for prompt-injection/jailbreak safeguarding.
|
|
- Two-tier auth: `key-auth` (static, shared key) for chatbot UIs that can't
|
|
do OIDC on their outbound LLM connection (LibreChat and similar tools —
|
|
structural limitation of that whole category, not fixable client-side),
|
|
`openid-connect` for agents built in-house. Either succeeds; a request
|
|
passing neither is rejected (`request-termination` on the fallback
|
|
anonymous consumer).
|
|
|
|
**Not yet active**: PII redaction (`ai-sanitizer`) — needs a private Kong
|
|
Cloudsmith registry image and an AI Gateway Enterprise entitlement, blocked
|
|
pending that access. Everything else in this repo is unaffected.
|
|
|
|
## Structure
|
|
|
|
- `dev/ai-gateway-kustomization.yaml` + `dev/ai-gateway-externalsecret.yaml`
|
|
— Gateway/DataPlane/Control Plane infra (same shared `k8s-platform`
|
|
template every domain uses).
|
|
- `dev/proxy-kustomization.yaml` + `dev/proxy-externalsecret.yaml` — the
|
|
LLM proxy itself (`./kong`).
|
|
- `dev/gitrepository.yaml` — this repo, self-referential.
|