Stéphane Tailland
05cf2bcce1
Four settings, all of them off, which is why the AI gateway produced
HTTP-level telemetry and nothing about the model.
logging.log_payloads the prompt and the completion, in the log
records and — since OTel instrumentation is on
here — in the spans too
logging.log_statistics model usage and token counts
prometheus.ai_metrics kong_ai_llm_* : requests, tokens by type, cost,
provider latency, per model and per consumer
input_cost / output_cost cost per 1M tokens, which is what makes
kong_ai_llm_cost_total mean anything
Also removes this morning's pre-function/post-function body capture from
this gateway. It was the wrong tool for the LLM route: Kong's own
implementation reassembles a streamed SSE answer, while accumulating raw
chunks in kong.ctx.shared does not. The five business domains keep theirs
— their MCP traffic is not streamed the same way.
The costs are a snapshot, checked against OpenRouter's price list for
google/gemini-3.7-flash on 2026-08-27: $0.375 in, $1.875 out per 1M
tokens. They are substitution variables rather than literals precisely
because a stale figure produces confidently wrong money rather than an
error.
Note what this does NOT fix: usage is missing on streamed responses
because the OpenAI-compatible API only reports it when the caller asks
for stream_options.include_usage, and that is LibreChat's request to
make, not Kong's. Expect token counts to stay partial until then.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
38 lines
1.5 KiB
YAML
38 lines
1.5 KiB
YAML
apiVersion: configuration.konghq.com/v1
|
|
kind: KongPlugin
|
|
metadata:
|
|
name: ${NAME}-llm-ai-proxy-advanced
|
|
namespace: ${NAMESPACE}
|
|
plugin: ai-proxy-advanced
|
|
config:
|
|
targets:
|
|
- route_type: llm/v1/chat
|
|
auth:
|
|
header_name: Authorization
|
|
header_value: ${OPENROUTER_BEARER}
|
|
# dev only, like the rest of this gateway's audit trail: log_payloads
|
|
# puts the prompt and the completion in the log stream, and -- because
|
|
# OpenTelemetry instrumentation is on here -- in the spans as well.
|
|
#
|
|
# This is what the pre-function/post-function capture in
|
|
# observability.yaml stood in for on this route. Kong's own
|
|
# implementation reassembles a streamed SSE answer; accumulating raw
|
|
# chunks does not.
|
|
logging:
|
|
log_payloads: true
|
|
# Model usage and token counts in the log records. With this off Kong
|
|
# counted nothing of its own, which is half of why usage was missing.
|
|
log_statistics: true
|
|
model:
|
|
provider: openai
|
|
# name deliberately unset: the client's own `model` field is passed
|
|
# through to OpenRouter.
|
|
options:
|
|
upstream_url: ${OPENROUTER_UPSTREAM_URL}
|
|
# Cost per 1M tokens, in the provider's currency. Kong multiplies
|
|
# these by the token counts to produce kong_ai_llm_cost_total, so a
|
|
# stale figure yields confidently wrong money rather than an error.
|
|
# Re-check them whenever the model or the contract changes.
|
|
input_cost: ${LLM_INPUT_COST}
|
|
output_cost: ${LLM_OUTPUT_COST}
|