The proxy cache held every GET on the whole service for 60 seconds, and these
APIs accept writes. Kong's proxy cache has no invalidation on write: a POST
that changes a product passes through the same gateway, and the gateway goes on
serving the value it captured before it, for up to a minute, with nothing in the
response saying so.
It produced a real wrong answer. An agent woken by a stock event read the figure
from before that event and reported, in one sentence, "68 in stock, above the
threshold" and "it crossed the threshold, from 68".
The manifest is kept, unreferenced, carrying the reason — so reinstating it is a
decision rather than a rediscovery.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ACL answers 'may this identity call create-product'. Nothing answered 'may
it call it with these arguments' — write authorises writing, not this
particular record. And the read cap counts calls, so one call asking for a
hundred thousand rows cost the same as one asking for ten.
The OPA plugin sends the decoded JSON-RPC body, so the decision point sees the
tool name and its arguments. The policy lives once in k8s-platform/opa and every
domain asks the same server.
Enforced at the gateway rather than in an agent, because an agent that has been
talked into something is the case this exists for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A scope authorises a read and says nothing about volume. An agent reads a
thousand records in seconds where a person clicks ten, and until now nothing
noticed: the token is valid, the tool is permitted, the ACL passes. That gap
stopped being theoretical the day an autonomous run started reading the whole
catalog on a schedule.
identifier: credential keys the counter on the sub claim — the end user,
whoever is asking on their behalf — so an agent looping under one person's
identity spends that person's budget and cannot spend everyone's. Verified on
catalog: pm-user and ro-user hold separate counters.
Per-route namespace, so a busy domain cannot exhaust another's.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The response schemas added yesterday made every paginated list tool fail. The
plugin's fields are documented as OpenAPI format and validated with a JSON
Schema validator, and `nullable: true` is not a JSON Schema keyword — so it is
ignored, `type: string` is enforced, and `previous: null` on the first page is
rejected as *null is not of type "string"*.
The failure lands after the backend has already done the work, surfaces to the
caller as a bare tool retry exhaustion, and appears nowhere in the gateway's own
logs. Found only because an orchestrating agent asked a question that paged.
The generator now rewrites `nullable: true` into `type: [T, "null"]` wherever it
inlines a schema, request bodies included. A 3.1 spec needs none of this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Produced by k8s-platform/tools/generate-mcp-tools.py: deck file openapi2mcp
does the conversion, and the wrapper re-injects the five things Kong's
generator cannot emit — per-tool ACLs, the MCP behavioural annotations, the
response schemas the plugin uses for structuredOutput, the real upstream host,
and JSON-only request bodies. Each is a documented gap; see
k8s-platform/MCP-TOOL-GENERATION.md.
tools-policy.yaml carries the ACLs and doubles as the allow-list: an operation
becomes a tool only if the policy names it. Kong would otherwise expose every
operation in the spec, DELETE included, and can express no authorization at all
from a bearer-token spec — silently, exit 0, no warning.
Same tools as before, same ACLs. What changed: query parameters are now
declared and callable, each tool carries its response schema so calls return
structured content instead of raw JSON, and request bodies are no longer
tripled across three identical content types.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every agent has to name this client in the audience scopes it requests, so the
one thing an agent cannot avoid naming was the very thing it should know
nothing about. The gateway is replaceable; the contract should not say
otherwise.
token-exchange-broker, same credential, same behaviour. The old client stays in
the realm until every consumer has moved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dependsOn still named flux-system, where the gateway Kustomization used to
live. It moved into the domain namespace so the feature team can reach it,
and a dependency naming the old location resolves to nothing — Flux
reported "dependency not found" and held this Kustomization not ready
rather than failing loudly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dev/ declares this repo's GitRepository, the Kustomization that deploys
./kong, and its ExternalSecrets — but nothing watched dev/, so those files
were applied once by hand and never re-read. Editing them in git changed
nothing until someone remembered to kubectl apply, which is what silently
swallowed the CONTROL_PLANE_REF_NAME rollout: correct in git for hours
while the cluster still held the old value.
Add the bootstrap Kustomization that watches ./dev and, living in ./dev,
manages itself. This is what `flux bootstrap` would have created;
flux-operator installs the controllers and no root.
It sits in the domain namespace rather than flux-system, because sourceRef
resolves in the Kustomization's own namespace and because this is the
team's to own.
Needs one kubectl apply to exist. That is the last one this repo needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The developer portal listed this API with no reference documentation:
catalog was the only domain that had a spec anywhere in the tree.
Generated from the backend's own drf-spectacular schema
(/classic-models/api/schema/, version 4.7.0), filtered to the operations
this domain's gateway actually exposes — derived from the Kong routes and
their request-transformer rewrites in Konnect, not from guesswork. Every
route resolved to a backend operation; the counts match one for one.
Paths and server are the **gateway's**, not the backend's. A portal
reader calls https://customers-dev.sttlab.local/customers/v1/..., and the
gateway rewrites that onto /classic-models/api/v1/... upstream. Path
parameters are renamed to the gateway's capture names where the backend
used a different case.
Components are pruned to what these paths transitively reference.
Note: catalog-api/openapi.yaml does NOT follow this convention — it
documents the backend paths and http://localhost:8000 as its server, so
its operations cannot be called as written from the portal. Left alone
here; worth aligning separately.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CONTROL_PLANE_REF_NAME held the random suffix the Gateway operator gives
its auto-created mirror CR, looked up by hand. Replacing the Konnect org
recreated those CRs under new suffixes, so the ref named an object that
no longer existed and the domain's Kong entities sat Programmed=False.
Name the ${NAME}-cp mirror declared in k8s-platform's dataplane-template
instead — derived from NAME, so it stays valid across Gateway
recreations and Konnect org changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Post http-log to /customers instead of /logs. Fluent Bit's http input derives the
record tag from the request URI, so the path is what identifies this domain
in the collector and becomes the Loki label domain=customers.
The collector keeps writing every domain to the on-disk audit trail as
well, so this is reversible by restoring the previous path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The second ai-mcp-proxy binding (added to fix the GET/SSE 502 loop)
created a second runtime plugin instance with the same tool config,
which ai-mcp-proxy's own duplicate-name detection caught: all three
tools (list-customers/get-customer/create-customer) fell back to
auto-generated names across both instances. Confirmed via dataplane
pod logs right after applying the previous fix.
Real fix: one KongRoute for all three MCP HTTP methods (GET/POST/
DELETE) instead of two, one ai-mcp-proxy binding instead of two.
ai-mcp-proxy already tolerates multi-method routes (same "skips
method selection" notice the existing POST+DELETE route already
produces) — extending to GET is the same class of thing, not new
risk, and there is now exactly one plugin instance so no duplicate
tool names.
The stream route (GET, mcp/routes.yaml) had no ai-mcp-proxy binding,
so GET requests fell through to the KongService's own upstream
target (REST_HOSTNAME, i.e. Kong's own external hostname) and
looped back into the Gateway, growing response headers each pass
until nginx rejected them (502, "upstream sent too big header").
Confirmed via customers dataplane pod logs while debugging a
LibreChat connection failure. catalog-mcp has the same structural
gap but happens to 405 instead of looping there — not touched here.
8 routes over Customer + orders/payments sub-resources, proxying to
classic-models-api. openid-connect with RFC 8693 token exchange to
classic-models-api audience, rate-limiting, proxy-cache, audit-log —
same plugin set and structure as catalog-api's REST surface.
Not deployed: dev/gitrepository.yaml points at a gitea repo that
doesn't exist yet (kong-customers-api.git, 404). No MCP layer yet
either, unlike catalog-api.
Stéphane Tailland