Stéphane Tailland
05cf2bcce1
Four settings, all of them off, which is why the AI gateway produced
HTTP-level telemetry and nothing about the model.
logging.log_payloads the prompt and the completion, in the log
records and — since OTel instrumentation is on
here — in the spans too
logging.log_statistics model usage and token counts
prometheus.ai_metrics kong_ai_llm_* : requests, tokens by type, cost,
provider latency, per model and per consumer
input_cost / output_cost cost per 1M tokens, which is what makes
kong_ai_llm_cost_total mean anything
Also removes this morning's pre-function/post-function body capture from
this gateway. It was the wrong tool for the LLM route: Kong's own
implementation reassembles a streamed SSE answer, while accumulating raw
chunks in kong.ctx.shared does not. The five business domains keep theirs
— their MCP traffic is not streamed the same way.
The costs are a snapshot, checked against OpenRouter's price list for
google/gemini-3.7-flash on 2026-08-27: $0.375 in, $1.875 out per 1M
tokens. They are substitution variables rather than literals precisely
because a stale figure produces confidently wrong money rather than an
error.
Note what this does NOT fix: usage is missing on streamed responses
because the OpenAI-compatible API only reports it when the caller asks
for stream_options.include_usage, and that is LibreChat's request to
make, not Kong's. Expect token counts to stay partial until then.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
124 lines
4.0 KiB
YAML
124 lines
4.0 KiB
YAML
# Observability plugins for the AI gateway.
|
|
#
|
|
# The five business domains get these from k8s-platform's
|
|
# dataplane-globalpolicy-template, but that template also carries
|
|
# openid-connect, rate-limiting and cors — all of which this gateway already
|
|
# defines its own AI-specific versions of. Applying it here would duplicate
|
|
# them, so the three observability plugins are declared locally instead,
|
|
# with the same configuration as the shared template.
|
|
#
|
|
# That parity is manual and has already broken once: the propagation block
|
|
# below was added to the shared template and not here, so this gateway kept
|
|
# starting a fresh trace per request while the five domains propagated
|
|
# correctly. Any change to the shared template's observability plugins has to
|
|
# be mirrored here by hand.
|
|
apiVersion: configuration.konghq.com/v1
|
|
kind: KongPlugin
|
|
metadata:
|
|
name: ${NAME}-llm-prometheus
|
|
namespace: ${NAMESPACE}
|
|
plugin: prometheus
|
|
config:
|
|
# The reason this gateway's prometheus config differs from the shared
|
|
# template: it fronts an LLM. ai_metrics adds kong_ai_llm_* — requests,
|
|
# tokens by type, cost and provider latency, per model and per consumer.
|
|
# Without it none of that reaches Prometheus, whatever ai-proxy logs.
|
|
ai_metrics: true
|
|
status_code_metrics: true
|
|
latency_metrics: true
|
|
bandwidth_metrics: true
|
|
upstream_health_metrics: true
|
|
---
|
|
apiVersion: configuration.konghq.com/v1
|
|
kind: KongPlugin
|
|
metadata:
|
|
name: ${NAME}-llm-opentelemetry
|
|
namespace: ${NAMESPACE}
|
|
plugin: opentelemetry
|
|
config:
|
|
traces_endpoint: ${OTEL_TRACES_ENDPOINT}
|
|
resource_attributes:
|
|
service.name: ${NAME}
|
|
sampling_rate: 1
|
|
http_response_header_for_traceid: X-Trace-Id
|
|
# Without this block every hop starts its own trace: a single end-to-end
|
|
# transaction shows up in Tempo as a dozen unrelated traces, one per service.
|
|
# `extract` is what makes the gateway continue the caller's trace instead of
|
|
# opening a new one, and `inject` is what carries it to the upstream so the
|
|
# backend can continue it in turn.
|
|
#
|
|
# This gateway is where LibreChat enters, so without it the chain breaks at
|
|
# the very first hop and nothing downstream can be correlated.
|
|
propagation:
|
|
extract:
|
|
- w3c
|
|
- b3
|
|
- jaeger
|
|
inject:
|
|
- w3c
|
|
default_format: w3c
|
|
clear: []
|
|
---
|
|
apiVersion: configuration.konghq.com/v1
|
|
kind: KongPlugin
|
|
metadata:
|
|
name: ${NAME}-llm-audit-log
|
|
namespace: ${NAMESPACE}
|
|
# dev only: the raw Authorization header is captured in clear text, which is
|
|
# not acceptable as-is outside dev.
|
|
plugin: http-log
|
|
config:
|
|
http_endpoint: ${AUDIT_LOG_ENDPOINT}
|
|
method: POST
|
|
content_type: application/json
|
|
---
|
|
# prometheus and opentelemetry are bound control-plane-wide, matching the
|
|
# shared template: they describe the gateway itself, not one service.
|
|
apiVersion: configuration.konghq.com/v1alpha1
|
|
kind: KongPluginBinding
|
|
metadata:
|
|
name: ${NAME}-llm-prometheus-binding
|
|
namespace: ${NAMESPACE}
|
|
spec:
|
|
controlPlaneRef:
|
|
type: konnectNamespacedRef
|
|
konnectNamespacedRef:
|
|
name: ${CONTROL_PLANE_REF_NAME}
|
|
pluginRef:
|
|
name: ${NAME}-llm-prometheus
|
|
scope: GlobalInControlPlane
|
|
---
|
|
apiVersion: configuration.konghq.com/v1alpha1
|
|
kind: KongPluginBinding
|
|
metadata:
|
|
name: ${NAME}-llm-opentelemetry-binding
|
|
namespace: ${NAMESPACE}
|
|
spec:
|
|
controlPlaneRef:
|
|
type: konnectNamespacedRef
|
|
konnectNamespacedRef:
|
|
name: ${CONTROL_PLANE_REF_NAME}
|
|
pluginRef:
|
|
name: ${NAME}-llm-opentelemetry
|
|
scope: GlobalInControlPlane
|
|
---
|
|
# The audit log is bound to the proxy service, like the business domains bind
|
|
# theirs, so only real gateway traffic is captured.
|
|
apiVersion: configuration.konghq.com/v1alpha1
|
|
kind: KongPluginBinding
|
|
metadata:
|
|
name: ${NAME}-llm-audit-log-binding
|
|
namespace: ${NAMESPACE}
|
|
spec:
|
|
controlPlaneRef:
|
|
type: konnectNamespacedRef
|
|
konnectNamespacedRef:
|
|
name: ${CONTROL_PLANE_REF_NAME}
|
|
pluginRef:
|
|
name: ${NAME}-llm-audit-log
|
|
targets:
|
|
serviceRef:
|
|
group: configuration.konghq.com
|
|
kind: KongService
|
|
name: ${NAME}-llm-proxy
|