Add metrics, traces and audit logs to the AI gateway

This gateway had none of the three: no prometheus, opentelemetry or http-log
plugin, so it produced only Kong's own node and shared-dict metrics and was
invisible in Tempo and Loki.

The five business domains inherit these from k8s-platform's
dataplane-globalpolicy-template, but that template also carries
openid-connect, rate-limiting and cors, and this gateway already defines its
own AI-specific versions of all three. Applying it here would duplicate them,
so the observability plugins are declared locally with the same configuration
as the shared template.

prometheus and opentelemetry bind control-plane-wide since they describe the
gateway itself; the audit log binds to the proxy service, matching how the
domains bind theirs.

The trace endpoint now comes from the shared dataplane-template-defaults
ConfigMap rather than being repeated here, so a change of trace backend does
not need a second edit. Its only key overlapping the inline block is
OIDC_ISSUER, and both carry the same value.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dev/proxy-kustomization.yaml

@@ -31,6 +31,17 @@ spec:
       # username (Konnect stores it as a literal string otherwise, silently
       # never resolving — verified live, request-termination never fired).
       LLM_ANONYMOUS_CONSUMER_ID: "1efd79ee-fd2d-4f2d-beb8-e6e9b24185a1"
+      # The URI path is the collector's routing key: Fluent Bit tags the
+      # record `llm` and turns that into the Loki label domain=llm.
+      AUDIT_LOG_ENDPOINT: http://audit-log-collector.observability.svc.cluster.local/llm
     substituteFrom:
+      # Listed first so the Secret below still wins on any key it defines.
+      # Brings OTEL_TRACES_ENDPOINT, the cluster-wide trace backend, so this
+      # gateway follows a backend change without a second edit here. The only
+      # key it shares with the inline block above is OIDC_ISSUER, and both
+      # carry the same value.
+      - kind: ConfigMap
+        name: dataplane-template-defaults
+        optional: false
       - kind: Secret
         name: platform-llm-dev-secrets

kong/kustomization.yaml

@@ -13,3 +13,4 @@ resources:
   - openid-connect.yaml
   - consumer.yaml
   - pluginbindings.yaml
+  - observability.yaml

kong/observability.yaml

@@ -0,0 +1,95 @@
+# Observability plugins for the AI gateway.
+#
+# The five business domains get these from k8s-platform's
+# dataplane-globalpolicy-template, but that template also carries
+# openid-connect, rate-limiting and cors — all of which this gateway already
+# defines its own AI-specific versions of. Applying it here would duplicate
+# them, so the three observability plugins are declared locally instead,
+# with the same configuration as the shared template.
+apiVersion: configuration.konghq.com/v1
+kind: KongPlugin
+metadata:
+  name: ${NAME}-llm-prometheus
+  namespace: ${NAMESPACE}
+plugin: prometheus
+config:
+  status_code_metrics: true
+  latency_metrics: true
+  bandwidth_metrics: true
+  upstream_health_metrics: true
+---
+apiVersion: configuration.konghq.com/v1
+kind: KongPlugin
+metadata:
+  name: ${NAME}-llm-opentelemetry
+  namespace: ${NAMESPACE}
+plugin: opentelemetry
+config:
+  traces_endpoint: ${OTEL_TRACES_ENDPOINT}
+  resource_attributes:
+    service.name: ${NAME}
+  sampling_rate: 1
+  http_response_header_for_traceid: X-Trace-Id
+---
+apiVersion: configuration.konghq.com/v1
+kind: KongPlugin
+metadata:
+  name: ${NAME}-llm-audit-log
+  namespace: ${NAMESPACE}
+# dev only: the raw Authorization header is captured in clear text, which is
+# not acceptable as-is outside dev.
+plugin: http-log
+config:
+  http_endpoint: ${AUDIT_LOG_ENDPOINT}
+  method: POST
+  content_type: application/json
+---
+# prometheus and opentelemetry are bound control-plane-wide, matching the
+# shared template: they describe the gateway itself, not one service.
+apiVersion: configuration.konghq.com/v1alpha1
+kind: KongPluginBinding
+metadata:
+  name: ${NAME}-llm-prometheus-binding
+  namespace: ${NAMESPACE}
+spec:
+  controlPlaneRef:
+    type: konnectNamespacedRef
+    konnectNamespacedRef:
+      name: ${CONTROL_PLANE_REF_NAME}
+  pluginRef:
+    name: ${NAME}-llm-prometheus
+  scope: GlobalInControlPlane
+---
+apiVersion: configuration.konghq.com/v1alpha1
+kind: KongPluginBinding
+metadata:
+  name: ${NAME}-llm-opentelemetry-binding
+  namespace: ${NAMESPACE}
+spec:
+  controlPlaneRef:
+    type: konnectNamespacedRef
+    konnectNamespacedRef:
+      name: ${CONTROL_PLANE_REF_NAME}
+  pluginRef:
+    name: ${NAME}-llm-opentelemetry
+  scope: GlobalInControlPlane
+---
+# The audit log is bound to the proxy service, like the business domains bind
+# theirs, so only real gateway traffic is captured.
+apiVersion: configuration.konghq.com/v1alpha1
+kind: KongPluginBinding
+metadata:
+  name: ${NAME}-llm-audit-log-binding
+  namespace: ${NAMESPACE}
+spec:
+  controlPlaneRef:
+    type: konnectNamespacedRef
+    konnectNamespacedRef:
+      name: ${CONTROL_PLANE_REF_NAME}
+  pluginRef:
+    name: ${NAME}-llm-audit-log
+  targets:
+    serviceRef:
+      group: configuration.konghq.com
+      kind: KongService
+      name: ${NAME}-llm-proxy