Stéphane Tailland
05cf2bcce1
Four settings, all of them off, which is why the AI gateway produced
HTTP-level telemetry and nothing about the model.
logging.log_payloads the prompt and the completion, in the log
records and — since OTel instrumentation is on
here — in the spans too
logging.log_statistics model usage and token counts
prometheus.ai_metrics kong_ai_llm_* : requests, tokens by type, cost,
provider latency, per model and per consumer
input_cost / output_cost cost per 1M tokens, which is what makes
kong_ai_llm_cost_total mean anything
Also removes this morning's pre-function/post-function body capture from
this gateway. It was the wrong tool for the LLM route: Kong's own
implementation reassembles a streamed SSE answer, while accumulating raw
chunks in kong.ctx.shared does not. The five business domains keep theirs
— their MCP traffic is not streamed the same way.
The costs are a snapshot, checked against OpenRouter's price list for
google/gemini-3.7-flash on 2026-08-27: $0.375 in, $1.875 out per 1M
tokens. They are substitution variables rather than literals precisely
because a stale figure produces confidently wrong money rather than an
error.
Note what this does NOT fix: usage is missing on streamed responses
because the OpenAI-compatible API only reports it when the caller asks
for stream_options.include_usage, and that is LibreChat's request to
make, not Kong's. Expect token counts to stay partial until then.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
59 lines
2.7 KiB
YAML
59 lines
2.7 KiB
YAML
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform-llm-dev
|
|
namespace: llm
|
|
spec:
|
|
interval: 10m
|
|
sourceRef:
|
|
kind: GitRepository
|
|
name: platform-llm-dev
|
|
path: ./kong
|
|
prune: true
|
|
dependsOn:
|
|
- name: llm-dev-ai-gateway
|
|
namespace: flux-system
|
|
postBuild:
|
|
substitute:
|
|
NAME: llm-dev
|
|
NAMESPACE: llm
|
|
CONTROL_PLANE_REF_NAME: llm-dev-cp
|
|
OIDC_ISSUER: https://keycloak.sttlab.eu/realms/sttlab
|
|
AGENT_AUDIENCE: '["llm-proxy"]'
|
|
OPENROUTER_UPSTREAM_URL: https://openrouter.ai/api/v1/chat/completions
|
|
# Cost per 1M tokens for the model behind OPENROUTER_UPSTREAM_URL, used by
|
|
# ai-proxy-advanced to compute kong_ai_llm_cost_total. Checked against
|
|
# OpenRouter's price list for google/gemini-3.7-flash on 2026-08-27
|
|
# ($0.375 in / $1.875 out). These are a snapshot, not a live rate: change
|
|
# the model or the contract and they silently become wrong money.
|
|
LLM_INPUT_COST: "0.375"
|
|
LLM_OUTPUT_COST: "1.875"
|
|
OPENROUTER_EMBEDDINGS_UPSTREAM_URL: https://openrouter.ai/api/v1/embeddings
|
|
VECTORDB_HOST: llm-dev-vectordb
|
|
VECTORDB_PORT: "6379"
|
|
RATE_LIMIT_GLOBAL_TOKENS_PER_MINUTE: "500000"
|
|
RATE_LIMIT_PER_CLIENT_TOKENS_PER_MINUTE: "50000"
|
|
# Looked up after consumer.yaml's llm-anonymous KongConsumer first
|
|
# reconciled — config.anonymous needs the real Consumer ID, not the
|
|
# username (Konnect stores it as a literal string otherwise, silently
|
|
# never resolving — verified live, request-termination never fired).
|
|
LLM_ANONYMOUS_CONSUMER_ID: "1efd79ee-fd2d-4f2d-beb8-e6e9b24185a1"
|
|
# The URI path is the collector's routing key: Fluent Bit tags the
|
|
# record `llm` and turns that into the Loki label domain=llm.
|
|
AUDIT_LOG_ENDPOINT: http://audit-log-collector.observability.svc.cluster.local/llm
|
|
# Duplicated from k8s-platform's dataplane-template-defaults ConfigMap
|
|
# rather than pulled via substituteFrom, because that ConfigMap is
|
|
# deliberately single-instance in flux-system and this Kustomization
|
|
# cannot live there: its substitution Secret below holds real
|
|
# credentials (OPENROUTER_BEARER, LIBRECHAT_APIKEY), and putting those
|
|
# in flux-system is the exact thing the Konnect PAT comment in
|
|
# gateway-kustomization.yaml refuses to do. Referencing the ConfigMap
|
|
# from the llm namespace just fails the build outright
|
|
# (optional: false), which is why this stack never deployed.
|
|
# Cost of the duplication: changing the cluster trace backend now takes
|
|
# an edit here too.
|
|
OTEL_TRACES_ENDPOINT: http://tempo.observability.svc.cluster.local:4318/v1/traces
|
|
substituteFrom:
|
|
- kind: Secret
|
|
name: platform-llm-dev-secrets
|