catalog was the only domain that split its MCP route in two: the main
route on POST/DELETE, and a separate -stream route matching GET, whose
only purpose was to carry a request-termination plugin answering 405
"SSE stream is not offered on this endpoint". The other four domains
keep one route with GET, POST and DELETE.
That opt-out is not a fix the other four missed — it gives up a feature
the plugin implements. ai-mcp-proxy handles non-POST itself: its handler
carries an explicit non-POST branch, MCP session handling, an SSE
subscription waiting on notifications/tools/list_changed, and Accept
header validation for application/json and text/event-stream. GET is the
server-to-client channel of MCP Streamable HTTP, and on the other four
domains it reaches the plugin and works.
Nothing in git or in any comment records why catalog gave it up, and the
plugin and binding behind it were applied by hand three days after the
route landed — they were never in git at all. So on a cluster rebuilt
from git the split had no guard behind it: GET fell through to a route
with no plugin whose service points back at the gateway itself.
Drop the stream route and put GET back on the MCP route, so the five
domains are identical again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CONTROL_PLANE_REF_NAME held the random suffix the Gateway operator gives
its auto-created mirror CR, looked up by hand. Replacing the Konnect org
recreated those CRs under new suffixes, so the ref named an object that
no longer existed and the domain's Kong entities sat Programmed=False.
Name the ${NAME}-cp mirror declared in k8s-platform's dataplane-template
instead — derived from NAME, so it stays valid across Gateway
recreations and Konnect org changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Post http-log to /catalog instead of /logs. Fluent Bit's http input derives the
record tag from the request URI, so the path is what identifies this domain
in the collector and becomes the Loki label domain=catalog.
The collector keeps writing every domain to the on-disk audit trail as
well, so this is reversible by restoring the previous path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
metadata.name must be an RFC 1123 subdomain (no underscores); spec.name
(the value actually matched against the JWT role claim) stays read_only/
product_manager to match the realm role names exactly.
KongConsumerGroups were still named sttlab-user/sttlab-admin, a leftover
from before the classic-models realm's role model existed. Real users'
JWTs carry realm roles (admin, read_only, product_manager, customer,
support) via consumer_groups_claim, so no group ever matched and every
tool call was rejected with 403 "consumer groups were not found".
Groups now match the realm roles directly; tool ACLs gate on each role's
catalog:read/catalog:write grant from security-model.yaml.
Body capture lives in dedicated audit-log-body-capture(-bindings).yaml
files, separate from audit-log.yaml itself, so they can be dropped
without touching the base logging plugin (degrades gracefully to
metadata-only via nil custom_fields). Response body captured in
body_filter (chunk-accumulated) rather than the log phase, which
cannot call kong.response.get_raw_body() directly.
kong.request.get_raw_body()/kong.response.get_raw_body() cannot be
called from the log phase (confirmed via dataplane error logs:
"function cannot be called in log phase") — this silently killed
log_by_lua for every plugin on the request, not just body capture.
Back to metadata-only logging while body capture gets redone properly
(pre-function/post-function storing into kong.ctx.shared, read back
in custom_fields_by_lua).
custom_fields_by_lua adds request_body/response_body to the http-log
payload. Bodies and the raw Authorization header are now both logged
in clear text — dev only, flagged in the plugin file itself.
Turned out unnecessary: the real blocker was the missing
konghq.com/secret=true label on the Vault-sourced Secret, not a gap
in the Gateway API TLS translation itself. Once that label was added
(k8s-platform), the operator's own auto-generated KongCertificate/SNI
started working and this manual one just collided with it.
Gateway API's listeners[].tls.certificateRefs never gets synced to
Konnect by the gateway operator in this setup — same working pattern
as every other resource here (KongService, KongRoute, ...): a
Kong-native CRD, not the Gateway API's own translation.
Stéphane Tailland