feat: make the write path a mechanism instead of a prompt rule

The agent asked for confirmation because the system prompt told it to, and read
the answer itself. Both halves are now enforced.

Each MCP toolset is wrapped in ApprovalRequiredToolset whose predicate reads the
readOnlyHint the server publishes — the gateway decides what counts as a write,
and the agent keeps no list of tool names in step with it.
output_type=[str, DeferredToolRequests] lets a turn end holding a proposed call
instead of making it; the executor parks the A2A task in input-required and
renders the arguments the model actually sent, not what it said it would do.
The reply is matched against a closed set of words, so "oui mais change le prix"
denies the call and the correction comes back for approval on its own merits —
verified end to end, along with a draft the user approved that was still refused
because msrp sat below buyprice.

Also wired: a ProductDraft validated after approval and before the call leaves;
UsageLimits on every run; FilteredToolset denying sales' write tools inside the
agent as well as at the gateway; PrefixedToolset namespacing the two servers;
dynamic instructions naming the caller; Agent.instrument_all() for a span per
model call and per tool call; and pydantic_ai.retries on inference only, so a
429 from the gateway is waited out rather than failing mid-reasoning.

Inference gets its own httpx client: validate_response raises on any 4xx, and
applying that to the shared client would turn a token-exchange refusal into an
exception carrying no body — which is where Keycloak says what was wrong. Its
transport is built explicitly, because a transport passed to the client makes
verify= inert and the lab CA would otherwise go undeclared.

Tests grew from 7 to 31, covering what approval decides and what a draft must
satisfy. evals/catalogue_evals.py adds a behavioural suite, kept out of pytest
because every case spends real tokens; its first case checks that a count comes
from the tool rather than from what the model remembers of a public dataset.

CLAUDE.md corrected: instrument=True is not an Agent parameter in 2.35.3,
pydantic_ai.retries needs tenacity, TemplateStr needs pydantic-handlebars, and
approval does not remove the in-memory history — resuming still needs the
messages, so replicas: 1 stands until they live in the A2A task store.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Stéphane Tailland
2026-08-29 09:15:33 +02:00
parent aace433d7e
commit 1a13743078
12 changed files with 815 additions and 49 deletions
+99 -7
View File
@@ -1,6 +1,6 @@
# Catalogue — product management agent
**Framework:** PydanticAI + FastA2A · **Mode:** conversational · **Client id:** `classic-models-agent-catalogue`
**Framework:** PydanticAI + `a2a-sdk` · **Mode:** conversational · **Client id:** `classic-models-agent-catalogue`
Serves the product manager: create and update products and product lines, set
prices, decide what to discontinue.
@@ -32,14 +32,106 @@ constraints** — prices, quantities, references, product line membership. Typed
validation of the model's output earns its place here in a way it would not on a
purely conversational agent.
## FastA2A, not to_a2a()
**That argument is not yet cashed in.** `agent.py` builds `Agent(model,
instructions=..., toolsets=...)` with no `output_type`, no output validator and
no local tool, so the output is a free `str`. On the code as it stands,
PydanticAI is a bare MCP client and nothing more. Either wire the validation
(see below) or stop claiming it as the reason for the framework.
Use FastA2A directly. `Agent.to_a2a()` still works in PydanticAI 1.x but emits
a deprecation warning and is removed in v2.
## What is wired, and what it cost
FastA2A is framework-agnostic, so whatever is built here is also the natural
basis for the A2A adapter in front of the n8n agents — see
`classic-models-agent-flows`.
All of the below is in the code as of 2026-08-29. Verified against the installed
`pydantic-ai-slim` 2.35.3 — **three of these do add a dependency**, contrary to
what an earlier revision of this file claimed:
- `pydantic_ai.retries` ships in the package but imports `tenacity`, which the
slim extras do not pull. Added.
- `pydantic-evals` is a separate distribution. Added as a dev dependency.
- `TemplateStr` needs `pydantic-handlebars` (`pydantic-ai-slim[spec]`). **Not
added** — a template engine earns nothing against a single interpolation, so
the dynamic instructions are a plain function, which is the idiomatic API and
needs nothing.
1. **Approval as a mechanism, not a prompt rule.** Done, and it is the change
that matters. Each MCP toolset is wrapped in `ApprovalRequiredToolset`, whose
predicate reads the **`readOnlyHint` the server publishes** — so the gateway
decides what counts as a write and the agent keeps no list of tool names in
step with it. `output_type=[str, DeferredToolRequests]` lets a turn end
holding a proposed call instead of making it; the executor parks the task
with `TaskUpdater.requires_input()` and renders the arguments *the model
actually proposed*, not what it said it would do. The reply is read
deterministically against a closed set of words — a model must not be what
decides whether the user said yes, and "oui mais change le prix" is not
consent.
One claim in the earlier revision does not hold: this does **not** let the
in-memory history go. Resuming needs `message_history`, so the pending state
still lives in the process and `replicas: 1` still stands. Moving it into the
A2A task store is what would lift that, and it is not done.
2. **A validated draft on writes.** `approval.py` holds `ProductDraft` —
price above zero, stock within range, msrp above buyprice, no empty strings,
no extra fields. It runs *after* the user approves and *before* the call
leaves, so a record the user waved through is still refused if the model
built it wrong. Note this is argument validation, not `output_type`:
`output_type` describes a run's result, and the thing worth constraining
here is what goes out, not what comes back.
3. **Instrumentation.** `Agent.instrument_all()` at start-up, in
`__main__.py` — **not** `instrument=True` on `Agent(...)`, which does not
exist in 2.35.3; the parameters are `capabilities, defer_model_check,
deps_type, description, end_strategy, instructions, max_concurrency,
metadata, model, model_settings, name, output_type, retries, system_prompt,
tool_timeout, tools, toolsets, validation_context`. OpenTelemetry and
`logfire_api` are already present transitively. One span per model call and
per tool call: the MCP server records *which* tool ran, this records why.
4. **`UsageLimits(tool_calls_limit=...)`** on every run, from
`Settings.max_tool_calls` (20).
5. **Toolset shaping.** `FilteredToolset` denies sales' write tools inside the
agent as well as at the gateway — an admin reaching this agent would
otherwise be offered `create-order`. `PrefixedToolset` namespaces each
server, so a future collision between two domains is impossible rather than
silent.
6. **Dynamic instructions.** A `caller_instructions(ctx)` function appended
per run, naming the identity the agent is acting under. Deps-typed
(`CallerDeps`), no template engine — see above.
7. **`pydantic_ai.retries`** on inference only. `wait_retry_after` honours
the gateway's own `429 AI token rate limit exceeded`, so a turn survives the
limit instead of failing mid-reasoning. It gets a **separate httpx client**:
`validate_response` raises on any 4xx, and applying that to the shared client
would turn a token-exchange refusal into an exception carrying no body —
which is where Keycloak says what was wrong.
8. **`durable_exec`** — not done, on the reasoning in the earlier revision:
heavy, and the right fix for single-replica history is the A2A task store,
not a workflow engine.
**`pydantic-evals`**, now a dev dependency, with a suite in
`evals/catalogue_evals.py`. It is not part of `pytest`: every case spends real
tokens through the gateway and needs a live realm.
Its cases are the failures actually seen, not coverage. The first one is the
important one: an agent asked for a product count once answered the true figure
while its tool had returned a page holding a different one. Right answer, green
trace, ungrounded number — and only a check against the data tells the two
apart.
The unit tests grew to 31 alongside, covering what approval decides and what a
draft must satisfy. Every rule they check used to live in the system prompt,
which is to say it used to be a request.
Keep the split straight while doing any of this: none of it is authorisation.
Who may write is decided by the MCP server against the end user's own roles, at
discovery time. Everything above makes the agent predictable, not privileged.
## a2a-sdk, not FastA2A
An earlier revision of this file said FastA2A, on the grounds that
`Agent.to_a2a()` was going away in PydanticAI v2 — which has since happened, so
that argument no longer separates the two. The code uses the official
`a2a-sdk` (1.1.x): this estate exists to prove A2A interoperability across five
runtimes, and the reference implementation is the safer thing for four other
stacks to interoperate with.
It is framework-agnostic too, so it stays the natural basis for the A2A adapter
in front of the n8n agents — see `classic-models-agent-flows`.
## Before writing code