Stéphane Tailland
2d4debd12a
A tool-calling agent re-sends the whole tool result on every iteration of its loop, so a single question against a tool returning a full collection burns the old budget in three turns and answers 429 in the middle of reasoning. Pagination on those tools is the real fix; this buys room until then and stays well inside the 500k global ceiling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
68 lines
3.3 KiB
YAML
68 lines
3.3 KiB
YAML
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform-llm-dev
|
|
namespace: llm
|
|
spec:
|
|
interval: 10m
|
|
sourceRef:
|
|
kind: GitRepository
|
|
name: platform-llm-dev
|
|
path: ./kong
|
|
prune: true
|
|
dependsOn:
|
|
- name: llm-dev-ai-gateway
|
|
# The AI gateway Kustomization moved out of flux-system into this
|
|
# namespace. A dependsOn still naming flux-system resolves to nothing
|
|
# and holds this Kustomization not ready rather than failing loudly.
|
|
namespace: llm
|
|
postBuild:
|
|
substitute:
|
|
NAME: llm-dev
|
|
NAMESPACE: llm
|
|
CONTROL_PLANE_REF_NAME: llm-dev-cp
|
|
# The classic-models realm on the lab Keycloak, not the stale VPS realm
|
|
# inherited from dataplane-template-defaults. Every domain gateway
|
|
# overrides it the same way; this one was missed, so its openid-connect
|
|
# plugin validated against the wrong JWKS and answered 401 to every
|
|
# token — visible only as "invalid signature" in the gateway log.
|
|
OIDC_ISSUER: https://keycloak.sttlab.local/realms/classic-models
|
|
AGENT_AUDIENCE: '["llm-proxy"]'
|
|
OPENROUTER_UPSTREAM_URL: https://openrouter.ai/api/v1/chat/completions
|
|
# Cost per 1M tokens for the model behind OPENROUTER_UPSTREAM_URL, used by
|
|
# ai-proxy-advanced to compute kong_ai_llm_cost_total. Checked against
|
|
# OpenRouter's price list for google/gemini-3.7-flash on 2026-08-27
|
|
# ($0.375 in / $1.875 out). These are a snapshot, not a live rate: change
|
|
# the model or the contract and they silently become wrong money.
|
|
LLM_INPUT_COST: "0.375"
|
|
LLM_OUTPUT_COST: "1.875"
|
|
OPENROUTER_EMBEDDINGS_UPSTREAM_URL: https://openrouter.ai/api/v1/embeddings
|
|
VECTORDB_HOST: llm-dev-vectordb
|
|
VECTORDB_PORT: "6379"
|
|
RATE_LIMIT_GLOBAL_TOKENS_PER_MINUTE: "500000"
|
|
# Doubled from 50000. A tool-calling agent re-sends the whole tool
|
|
# result on every iteration of its loop, so one question against a
|
|
# tool that returns a full collection (list-products is ~23 KB) can
|
|
# burn the old budget in three turns and answer 429 mid-reasoning.
|
|
# Pagination on those tools is the real fix; this buys room until
|
|
# then, and stays well inside the global ceiling.
|
|
RATE_LIMIT_PER_CLIENT_TOKENS_PER_MINUTE: "100000"
|
|
# Looked up after consumer.yaml's llm-anonymous KongConsumer first
|
|
# reconciled — config.anonymous needs the real Consumer ID, not the
|
|
# username (Konnect stores it as a literal string otherwise, silently
|
|
# never resolving — verified live, request-termination never fired).
|
|
LLM_ANONYMOUS_CONSUMER_ID: "1efd79ee-fd2d-4f2d-beb8-e6e9b24185a1"
|
|
# The URI path is the collector's routing key: Fluent Bit tags the
|
|
# record `llm` and turns that into the Loki label domain=llm.
|
|
AUDIT_LOG_ENDPOINT: http://audit-log-collector.observability.svc.cluster.local/llm
|
|
substituteFrom:
|
|
# Readable again now that k8s-platform places this ConfigMap in every
|
|
# domain namespace. It used to be flux-system only, which is why this
|
|
# Kustomization — which must stay here, holding real credentials — had to
|
|
# inline OTEL_TRACES_ENDPOINT instead. Listed first so the Secret still
|
|
# wins on any key it defines.
|
|
- kind: ConfigMap
|
|
name: dataplane-template-defaults
|
|
- kind: Secret
|
|
name: platform-llm-dev-secrets
|