feat: make the write path a mechanism instead of a prompt rule
The agent asked for confirmation because the system prompt told it to, and read the answer itself. Both halves are now enforced. Each MCP toolset is wrapped in ApprovalRequiredToolset whose predicate reads the readOnlyHint the server publishes — the gateway decides what counts as a write, and the agent keeps no list of tool names in step with it. output_type=[str, DeferredToolRequests] lets a turn end holding a proposed call instead of making it; the executor parks the A2A task in input-required and renders the arguments the model actually sent, not what it said it would do. The reply is matched against a closed set of words, so "oui mais change le prix" denies the call and the correction comes back for approval on its own merits — verified end to end, along with a draft the user approved that was still refused because msrp sat below buyprice. Also wired: a ProductDraft validated after approval and before the call leaves; UsageLimits on every run; FilteredToolset denying sales' write tools inside the agent as well as at the gateway; PrefixedToolset namespacing the two servers; dynamic instructions naming the caller; Agent.instrument_all() for a span per model call and per tool call; and pydantic_ai.retries on inference only, so a 429 from the gateway is waited out rather than failing mid-reasoning. Inference gets its own httpx client: validate_response raises on any 4xx, and applying that to the shared client would turn a token-exchange refusal into an exception carrying no body — which is where Keycloak says what was wrong. Its transport is built explicitly, because a transport passed to the client makes verify= inert and the lab CA would otherwise go undeclared. Tests grew from 7 to 31, covering what approval decides and what a draft must satisfy. evals/catalogue_evals.py adds a behavioural suite, kept out of pytest because every case spends real tokens; its first case checks that a count comes from the tool rather than from what the model remembers of a public dataset. CLAUDE.md corrected: instrument=True is not an Agent parameter in 2.35.3, pydantic_ai.retries needs tenacity, TemplateStr needs pydantic-handlebars, and approval does not remove the in-memory history — resuming still needs the messages, so replicas: 1 stands until they live in the A2A task store. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Catalogue — product management agent
|
||||
|
||||
**Framework:** PydanticAI + FastA2A · **Mode:** conversational · **Client id:** `classic-models-agent-catalogue`
|
||||
**Framework:** PydanticAI + `a2a-sdk` · **Mode:** conversational · **Client id:** `classic-models-agent-catalogue`
|
||||
|
||||
Serves the product manager: create and update products and product lines, set
|
||||
prices, decide what to discontinue.
|
||||
@@ -32,14 +32,106 @@ constraints** — prices, quantities, references, product line membership. Typed
|
||||
validation of the model's output earns its place here in a way it would not on a
|
||||
purely conversational agent.
|
||||
|
||||
## FastA2A, not to_a2a()
|
||||
**That argument is not yet cashed in.** `agent.py` builds `Agent(model,
|
||||
instructions=..., toolsets=...)` with no `output_type`, no output validator and
|
||||
no local tool, so the output is a free `str`. On the code as it stands,
|
||||
PydanticAI is a bare MCP client and nothing more. Either wire the validation
|
||||
(see below) or stop claiming it as the reason for the framework.
|
||||
|
||||
Use FastA2A directly. `Agent.to_a2a()` still works in PydanticAI 1.x but emits
|
||||
a deprecation warning and is removed in v2.
|
||||
## What is wired, and what it cost
|
||||
|
||||
FastA2A is framework-agnostic, so whatever is built here is also the natural
|
||||
basis for the A2A adapter in front of the n8n agents — see
|
||||
`classic-models-agent-flows`.
|
||||
All of the below is in the code as of 2026-08-29. Verified against the installed
|
||||
`pydantic-ai-slim` 2.35.3 — **three of these do add a dependency**, contrary to
|
||||
what an earlier revision of this file claimed:
|
||||
|
||||
- `pydantic_ai.retries` ships in the package but imports `tenacity`, which the
|
||||
slim extras do not pull. Added.
|
||||
- `pydantic-evals` is a separate distribution. Added as a dev dependency.
|
||||
- `TemplateStr` needs `pydantic-handlebars` (`pydantic-ai-slim[spec]`). **Not
|
||||
added** — a template engine earns nothing against a single interpolation, so
|
||||
the dynamic instructions are a plain function, which is the idiomatic API and
|
||||
needs nothing.
|
||||
|
||||
1. **Approval as a mechanism, not a prompt rule.** Done, and it is the change
|
||||
that matters. Each MCP toolset is wrapped in `ApprovalRequiredToolset`, whose
|
||||
predicate reads the **`readOnlyHint` the server publishes** — so the gateway
|
||||
decides what counts as a write and the agent keeps no list of tool names in
|
||||
step with it. `output_type=[str, DeferredToolRequests]` lets a turn end
|
||||
holding a proposed call instead of making it; the executor parks the task
|
||||
with `TaskUpdater.requires_input()` and renders the arguments *the model
|
||||
actually proposed*, not what it said it would do. The reply is read
|
||||
deterministically against a closed set of words — a model must not be what
|
||||
decides whether the user said yes, and "oui mais change le prix" is not
|
||||
consent.
|
||||
|
||||
One claim in the earlier revision does not hold: this does **not** let the
|
||||
in-memory history go. Resuming needs `message_history`, so the pending state
|
||||
still lives in the process and `replicas: 1` still stands. Moving it into the
|
||||
A2A task store is what would lift that, and it is not done.
|
||||
2. **A validated draft on writes.** `approval.py` holds `ProductDraft` —
|
||||
price above zero, stock within range, msrp above buyprice, no empty strings,
|
||||
no extra fields. It runs *after* the user approves and *before* the call
|
||||
leaves, so a record the user waved through is still refused if the model
|
||||
built it wrong. Note this is argument validation, not `output_type`:
|
||||
`output_type` describes a run's result, and the thing worth constraining
|
||||
here is what goes out, not what comes back.
|
||||
3. **Instrumentation.** `Agent.instrument_all()` at start-up, in
|
||||
`__main__.py` — **not** `instrument=True` on `Agent(...)`, which does not
|
||||
exist in 2.35.3; the parameters are `capabilities, defer_model_check,
|
||||
deps_type, description, end_strategy, instructions, max_concurrency,
|
||||
metadata, model, model_settings, name, output_type, retries, system_prompt,
|
||||
tool_timeout, tools, toolsets, validation_context`. OpenTelemetry and
|
||||
`logfire_api` are already present transitively. One span per model call and
|
||||
per tool call: the MCP server records *which* tool ran, this records why.
|
||||
4. **`UsageLimits(tool_calls_limit=...)`** on every run, from
|
||||
`Settings.max_tool_calls` (20).
|
||||
5. **Toolset shaping.** `FilteredToolset` denies sales' write tools inside the
|
||||
agent as well as at the gateway — an admin reaching this agent would
|
||||
otherwise be offered `create-order`. `PrefixedToolset` namespaces each
|
||||
server, so a future collision between two domains is impossible rather than
|
||||
silent.
|
||||
6. **Dynamic instructions.** A `caller_instructions(ctx)` function appended
|
||||
per run, naming the identity the agent is acting under. Deps-typed
|
||||
(`CallerDeps`), no template engine — see above.
|
||||
7. **`pydantic_ai.retries`** on inference only. `wait_retry_after` honours
|
||||
the gateway's own `429 AI token rate limit exceeded`, so a turn survives the
|
||||
limit instead of failing mid-reasoning. It gets a **separate httpx client**:
|
||||
`validate_response` raises on any 4xx, and applying that to the shared client
|
||||
would turn a token-exchange refusal into an exception carrying no body —
|
||||
which is where Keycloak says what was wrong.
|
||||
8. **`durable_exec`** — not done, on the reasoning in the earlier revision:
|
||||
heavy, and the right fix for single-replica history is the A2A task store,
|
||||
not a workflow engine.
|
||||
|
||||
**`pydantic-evals`**, now a dev dependency, with a suite in
|
||||
`evals/catalogue_evals.py`. It is not part of `pytest`: every case spends real
|
||||
tokens through the gateway and needs a live realm.
|
||||
|
||||
Its cases are the failures actually seen, not coverage. The first one is the
|
||||
important one: an agent asked for a product count once answered the true figure
|
||||
while its tool had returned a page holding a different one. Right answer, green
|
||||
trace, ungrounded number — and only a check against the data tells the two
|
||||
apart.
|
||||
|
||||
The unit tests grew to 31 alongside, covering what approval decides and what a
|
||||
draft must satisfy. Every rule they check used to live in the system prompt,
|
||||
which is to say it used to be a request.
|
||||
|
||||
Keep the split straight while doing any of this: none of it is authorisation.
|
||||
Who may write is decided by the MCP server against the end user's own roles, at
|
||||
discovery time. Everything above makes the agent predictable, not privileged.
|
||||
|
||||
## a2a-sdk, not FastA2A
|
||||
|
||||
An earlier revision of this file said FastA2A, on the grounds that
|
||||
`Agent.to_a2a()` was going away in PydanticAI v2 — which has since happened, so
|
||||
that argument no longer separates the two. The code uses the official
|
||||
`a2a-sdk` (1.1.x): this estate exists to prove A2A interoperability across five
|
||||
runtimes, and the reference implementation is the safer thing for four other
|
||||
stacks to interoperate with.
|
||||
|
||||
It is framework-agnostic too, so it stays the natural basis for the A2A adapter
|
||||
in front of the n8n agents — see `classic-models-agent-flows`.
|
||||
|
||||
## Before writing code
|
||||
|
||||
|
||||
Reference in New Issue
Block a user