docs: record Kong's MCP tool-generation limits and our workarounds

The domains' tools: arrays are hand-maintained, and that has now cost two
defects an agent found the hard way — create-product could never succeed, and
list-products cannot paginate, which let a model answer a product count that
looked right and did not come from the data.

Five limits measured against decK v1.64.0 and Kong Gateway 3.15, each
reproduced rather than read: per-tool ACLs cannot express a consumer-group
model and fail silently; the four MCP behavioural annotations are not
generated; response schemas are not generated although the plugin consumes them
for structuredOutput; servers is taken as the upstream target; every content
type is copied and every operation exposed.

FastMCP run on the same spec is recorded as a comparison point — it closes
three of the five out of the box — with the reason we are not adopting it: it
would move per-tool authorization out of the realm and into application code.

Also records that MCP Composer and MCP Runner never shipped, so none of this is
waiting on an imminent release.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

MCP-TOOL-GENERATION.md

@@ -0,0 +1,226 @@
+# MCP tool generation — what Kong does, what it does not, and what we do about it
+
+The five domain gateways expose their REST APIs as MCP servers through the
+`ai-mcp-proxy` plugin. This file records how those tool definitions are produced,
+the limits of Kong's generator, and the workarounds each limit forces on us.
+
+It exists because those workarounds are indistinguishable from design choices
+once they are in the repo. Written down, each one is a measurement of a gap in
+the product, and each can be deleted the day Kong closes it.
+
+**Measured 2026-08-29** against decK v1.64.0, Kong Gateway 3.15, and
+`catalog-api/openapi.yaml` (the catalog domain's spec, extracted from the live
+backend). Every claim below was reproduced, not read in documentation.
+
+## How the tool definitions are produced today
+
+They are not produced. `<domain>-api/kong/mcp/ai-mcp-proxy.yaml` carries a
+hand-maintained `tools:` array, committed to git and edited by hand.
+
+The array was almost certainly generated once — every tool's
+`annotations.title` is its OpenAPI `summary` verbatim, and tool names are
+`operationId` with underscores turned to hyphens, which is exactly what
+`deck file openapi2mcp` produces. It then stopped being regenerated while the
+specs kept moving.
+
+That drift is not theoretical. Two defects found on 2026-08-28 and 2026-08-29,
+both invisible until an agent hit them:
+
+- **`create-product` could never succeed.** Its schema omitted `productcode`,
+  which the API requires. Omit it and the API answers
+  `{"productcode":["This field is required."]}`; supply it and `ai-mcp-proxy`
+  rejects the call first with *Additional properties are not allowed
+  ('productcode' was unexpected)*. No input worked.
+- **`list-products` cannot paginate.** The API accepts `page` and `page_size`;
+  the tool declares no parameters at all. The caller gets the first page, 50 of
+  110 products, with nothing to say so. An agent asked how many products are in
+  a product line answered *38* — the true figure — while the tool had returned
+  a page containing 22. The number did not come from the data. The dataset is
+  public and the model knows it.
+
+  This is the worst failure mode for an observable estate: the answer is right,
+  the trace shows a successful tool call, and nothing marks the number as
+  ungrounded.
+
+Kong's generator emits `page` and `page_size` correctly. The bug is drift, not
+Kong. What follows are the things Kong genuinely does not do.
+
+## The generator
+
+`deck file openapi2mcp` is the only generally available OAS → MCP tool in the
+Kong ecosystem, confirmed three ways:
+
+- the `ai-mcp-proxy` plugin schema (read from the compiled bytecode in the
+  gateway image) has no spec-ingestion field — `parameters`, `request_body` and
+  `responses` are per-tool OpenAPI *fragments* you supply;
+- the Konnect API returns 404 for every MCP-related path
+  (`v3/mcp-servers`, `v2/mcp-servers`, `v3/mcp`, `v3/ai`, `v3/agents`,
+  `v3/tools`, `v3/composer`, `v3/ai-composer`, `v3/mcp-composer`) with a token
+  that returns 200 on `v2/control-planes`, `v3/apis` and `v3/portals`;
+- **MCP Composer and MCP Runner never shipped.** Announced for early 2026, the
+  product page that carried the AI Composer now describes *Context Mesh, in
+  tech preview*, with a registration form and no GA date. MCP Runner is no
+  longer named anywhere. What did land from that roadmap is the MCP Registry —
+  tech preview since February 2026, and a catalogue of existing MCP servers,
+  not a generator.
+
+Caveat on the Konnect probe: the token used belongs to org
+`b4531006-…`, not `2254dc3b-…` where the six domains live. Entitlements can
+differ per org, so the 404s prove absence from that org, not absence from the
+product.
+
+What it gets right, on our spec: `operationId` → kebab-case tool name matching
+our convention, `summary` → `annotations.title`, `description` → description,
+path *and* query parameters (including the pagination we are missing today),
+and `requestBody` with `$ref`s resolved and inlined. Chained through
+`deck file kong2kic` it produces a `KongPlugin` — the estate's own format.
+
+## The limits
+
+### 1. Per-tool ACLs cannot express our authorization model, and fail silently
+
+Kong generates per-tool ACLs only from an `oauth2` security scheme carrying the
+`x-kong-mcp-acl` extension, keyed on **OAuth scopes**.
+
+Our authorization is realm roles → consumer groups
+(`consumer_groups_claim: [realm_access, roles]`), and our specs declare
+`ApiKeyAuth` and `JWTAuth` (http bearer). Neither is `oauth2`, so no ACL is
+produced — and no `include_consumer_groups` either.
+
+**Nothing warns.** Verified at `--verbose 3`: no error, no warning, exit 0. A
+regeneration would silently publish an MCP server with every operation open to
+every caller, `create-*` and `delete-*` included.
+
+This is the most dangerous of the five, because the failure is silent and the
+blast radius is the whole domain.
+
+Note that the *plugin* supports our model perfectly well — its schema carries
+`include_consumer_groups`, `acl_attribute_type`, `access_token_claim_field`,
+`acl_list_field` and `default_acl`. The gap is in the generator only, and it
+matters: the ACL is what makes the estate work. Measured today, `pm-user` sees
+`create-product` in `tools/list` and `ro-user` does not. Kong filters at
+**discovery**, so a read-only user's turn contains no write tool for a prompt
+injection to reach.
+
+### 2. MCP behavioural annotations are not generated
+
+Kong emits `annotations.title` and nothing else. The MCP specification defines
+four more — `readOnlyHint`, `idempotentHint`, `destructiveHint`,
+`openWorldHint` — and all four are mechanically derivable from the HTTP method.
+The committed configuration has them; a regeneration would silently drop them.
+
+They are not decoration. An MCP client reads `destructiveHint` to decide
+whether to ask its user before calling a tool.
+
+### 3. Response schemas are not generated, though the plugin consumes them
+
+The plugin schema documents a per-tool `responses` field: *"This specification
+will be used to validate the upstream response and map it back to the
+structuredOutput."*
+
+So `ai-mcp-proxy` can validate the backend's response and return structured MCP
+output. Our specs contain the `responses` needed. `openapi2mcp` does not emit
+them, and neither does our hand-written configuration. Every tool therefore
+returns raw JSON for the model to interpret, and a backend that changes shape
+is caught by nobody.
+
+### 4. `servers` is taken as the upstream target
+
+The generated Kong Service points at `catalog-dev.sttlab.local:443` — the
+gateway's own public hostname, because that is what the spec documents for the
+developer portal. Applied as-is, the MCP proxy would loop back into the gateway
+instead of reaching the REST service (`${REST_HOSTNAME}`).
+
+The generator has no notion of *documented URL* versus *upstream target*. Any
+spec written for a portal — which is to say any spec that is also API
+documentation — hits this.
+
+### 5. Every content type is copied, and every operation is exposed
+
+DRF declares `application/json`, `application/x-www-form-urlencoded` and
+`multipart/form-data` for each write operation. Kong inlines the identical
+schema three times. `create-product`'s definition triples in size, for a
+payload the plugin will send as JSON. There is no flag to restrict it.
+
+And generation is all-or-nothing: 56 tools across the five domains against the
+25 we expose today, `PUT`, `PATCH` and `DELETE` included. `x-kong-mcp-exclude`
+exists but lives *in the spec*, and our specs are extracted from the live
+backend — the exclusions would be overwritten at the next extraction.
+
+## The comparison point
+
+FastMCP 3.4.7, run on the same `catalog-api/openapi.yaml`:
+
+```
+outils exposés: 4 -> ['create_product', 'get_product', 'list_product_lines', 'list_products']
+create_product      readOnly=False idempotent=False destructive=False  outputSchema=oui
+get_product         readOnly=True  idempotent=True  destructive=False  outputSchema=oui
+list_products       readOnly=True  idempotent=True  destructive=False  outputSchema=oui
+```
+
+Allow-listing through `route_map_fn`, the four behavioural annotations through a
+six-line `mcp_component_fn` deriving them from the HTTP method, and **output
+schemas generated from the OpenAPI `responses`** with no work at all. Limits 2,
+3 and 5 do not exist there.
+
+This is a comparison point, not a proposal. FastMCP produces a standalone MCP
+server, so adopting it would not swap one generator for another — it would take
+Kong out of the MCP server role and put five new services in its place, moving
+per-tool authorization from the realm into application code in each of them.
+That is the one thing the current architecture gets right, and the reason the
+recommendation below keeps Kong.
+
+What no tool does, Kong and FastMCP alike, is produce **decision-shaped** tools.
+Every generator maps one operation to one tool. An agent asking "can I promise
+delivery on 15 March?" wants one tool, not eleven CRUD endpoints to orchestrate.
+FastMCP's `Tool.from_tool` allows composing higher-level tools by hand;
+generating them is nobody's feature yet.
+
+## Workarounds
+
+### In place today
+
+**The `tools:` array is hand-maintained.** That is the workaround, and it is
+also the cause of both defects above. `create-product`'s schema was repaired by
+hand on 2026-08-29 — and the repair itself was wrong: it omitted
+`quantityinstock` from `required`, because it was written from the `Product`
+response schema instead of `ProductRequest`. Kong's generator gets that right.
+The hand-maintained file is not a stable state; it is a debt accruing silently.
+
+### Proposed, not yet built
+
+Keep `openapi2mcp` as the generator and add one post-processing step that
+re-injects what it cannot produce:
+
+| Step | Closes | Source of truth |
+|---|---|---|
+| Inject per-tool `acl.allow` and `include_consumer_groups` | limit 1 | a per-domain policy file, `kong/mcp/tools-policy.yaml` |
+| Derive the four annotations from the HTTP method | limit 2 | the spec's method |
+| Copy `responses` into each tool | limit 3 | the spec |
+| Rewrite the Service host to `${REST_HOSTNAME}` | limit 4 | the domain's own variables |
+| Keep `application/json` only; drop operations absent from the policy | limit 5 | the policy file |
+
+The policy file doubles as the allow-list: an operation becomes a tool only if
+the policy grants it an ACL. Adding a tool is then a deliberate act, and the
+current 25-tool surface is preserved exactly rather than silently tripled.
+
+Each row is a line item against Kong. If a future release generates
+annotations, that row is deleted and nothing else changes.
+
+## Reproducing the measurements
+
+```bash
+cd <domain>-api
+deck file openapi2mcp -s openapi.yaml --no-id -o /tmp/gen.yaml   # generate
+deck file kong2kic     -s /tmp/gen.yaml                          # to KongPlugin
+deck file openapi2mcp -s openapi.yaml --no-id --verbose 3 -o /dev/null | grep -i warn
+```
+
+The plugin's own schema, if the field list is ever in doubt:
+
+```bash
+POD=$(kubectl -n catalog get pods -o name | grep dataplane | head -1)
+kubectl -n catalog exec ${POD#pod/} -- \
+  cat /usr/local/share/lua/5.1/kong/plugins/ai-mcp-proxy/schema.ljbc > schema.ljbc
+LC_ALL=C strings schema.ljbc | LC_ALL=C grep -E '^[a-z][a-z_]{2,30}$' | sort -u
+```

README.md

@@ -7,6 +7,13 @@ Per-domain gateway infrastructure (`Gateway`, `GatewayConfiguration`,
 `KonnectAPIAuthConfiguration`) does NOT live here — see the dedicated
 `dataplane-<domain>` repos, to avoid write contention across domains.
 
+## MCP-TOOL-GENERATION.md
+
+How the domains' MCP tool definitions are produced, the five limits of Kong's
+`deck file openapi2mcp` generator, and the workaround each one forces. Read it
+before editing any `<domain>-api/kong/mcp/ai-mcp-proxy.yaml` by hand — that file
+is hand-maintained today, and that is precisely the problem it documents.
+
 ## flux/
 
 The Flux Operator (`flux-system` namespace) that everything else in this