dependsOn still named flux-system, where the gateway Kustomization used to
live. It moved into the domain namespace so the feature team can reach it,
and a dependency naming the old location resolves to nothing — Flux
reported "dependency not found" and held this Kustomization not ready
rather than failing loudly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dev/ declares this repo's GitRepository, the Kustomization that deploys
./kong, and its ExternalSecrets — but nothing watched dev/, so those files
were applied once by hand and never re-read. Editing them in git changed
nothing until someone remembered to kubectl apply, which is what silently
swallowed the CONTROL_PLANE_REF_NAME rollout: correct in git for hours
while the cluster still held the old value.
Add the bootstrap Kustomization that watches ./dev and, living in ./dev,
manages itself. This is what `flux bootstrap` would have created;
flux-operator installs the controllers and no root.
It sits in the domain namespace rather than flux-system, because sourceRef
resolves in the Kustomization's own namespace and because this is the
team's to own.
Needs one kubectl apply to exist. That is the last one this repo needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This spec predated the other four and documented the wrong surface: the
paths were the backend's (/classic-models/api/v1/...) and the server was
http://localhost:8000. Neither is reachable from the developer portal,
so every operation it published failed as written — the gateway serves
/catalog/v1/... and rewrites onto the backend path upstream.
Regenerated the same way as the other four domains: from the backend's
drf-spectacular schema, filtered to the operations this domain's gateway
exposes, derived from the Kong routes and their request-transformer
rewrites in Konnect. All 14 REST routes resolved to a backend operation.
Same six paths as before, now expressed as the gateway's, with
https://catalog-dev.sttlab.local as the server and path parameters
renamed to the gateway's capture names.
The five published specs now follow one convention.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
catalog was the only domain that split its MCP route in two: the main
route on POST/DELETE, and a separate -stream route matching GET, whose
only purpose was to carry a request-termination plugin answering 405
"SSE stream is not offered on this endpoint". The other four domains
keep one route with GET, POST and DELETE.
That opt-out is not a fix the other four missed — it gives up a feature
the plugin implements. ai-mcp-proxy handles non-POST itself: its handler
carries an explicit non-POST branch, MCP session handling, an SSE
subscription waiting on notifications/tools/list_changed, and Accept
header validation for application/json and text/event-stream. GET is the
server-to-client channel of MCP Streamable HTTP, and on the other four
domains it reaches the plugin and works.
Nothing in git or in any comment records why catalog gave it up, and the
plugin and binding behind it were applied by hand three days after the
route landed — they were never in git at all. So on a cluster rebuilt
from git the split had no guard behind it: GET fell through to a route
with no plugin whose service points back at the gateway itself.
Drop the stream route and put GET back on the MCP route, so the five
domains are identical again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CONTROL_PLANE_REF_NAME held the random suffix the Gateway operator gives
its auto-created mirror CR, looked up by hand. Replacing the Konnect org
recreated those CRs under new suffixes, so the ref named an object that
no longer existed and the domain's Kong entities sat Programmed=False.
Name the ${NAME}-cp mirror declared in k8s-platform's dataplane-template
instead — derived from NAME, so it stays valid across Gateway
recreations and Konnect org changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Post http-log to /catalog instead of /logs. Fluent Bit's http input derives the
record tag from the request URI, so the path is what identifies this domain
in the collector and becomes the Loki label domain=catalog.
The collector keeps writing every domain to the on-disk audit trail as
well, so this is reversible by restoring the previous path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
metadata.name must be an RFC 1123 subdomain (no underscores); spec.name
(the value actually matched against the JWT role claim) stays read_only/
product_manager to match the realm role names exactly.
KongConsumerGroups were still named sttlab-user/sttlab-admin, a leftover
from before the classic-models realm's role model existed. Real users'
JWTs carry realm roles (admin, read_only, product_manager, customer,
support) via consumer_groups_claim, so no group ever matched and every
tool call was rejected with 403 "consumer groups were not found".
Groups now match the realm roles directly; tool ACLs gate on each role's
catalog:read/catalog:write grant from security-model.yaml.
Body capture lives in dedicated audit-log-body-capture(-bindings).yaml
files, separate from audit-log.yaml itself, so they can be dropped
without touching the base logging plugin (degrades gracefully to
metadata-only via nil custom_fields). Response body captured in
body_filter (chunk-accumulated) rather than the log phase, which
cannot call kong.response.get_raw_body() directly.
kong.request.get_raw_body()/kong.response.get_raw_body() cannot be
called from the log phase (confirmed via dataplane error logs:
"function cannot be called in log phase") — this silently killed
log_by_lua for every plugin on the request, not just body capture.
Back to metadata-only logging while body capture gets redone properly
(pre-function/post-function storing into kong.ctx.shared, read back
in custom_fields_by_lua).
custom_fields_by_lua adds request_body/response_body to the http-log
payload. Bodies and the raw Authorization header are now both logged
in clear text — dev only, flagged in the plugin file itself.
Turned out unnecessary: the real blocker was the missing
konghq.com/secret=true label on the Vault-sourced Secret, not a gap
in the Gateway API TLS translation itself. Once that label was added
(k8s-platform), the operator's own auto-generated KongCertificate/SNI
started working and this manual one just collided with it.
Gateway API's listeners[].tls.certificateRefs never gets synced to
Konnect by the gateway operator in this setup — same working pattern
as every other resource here (KongService, KongRoute, ...): a
Kong-native CRD, not the Gateway API's own translation.
Stéphane Tailland