Files

T

Stéphane Tailland 1a13743078 feat: make the write path a mechanism instead of a prompt rule

The agent asked for confirmation because the system prompt told it to, and read
the answer itself. Both halves are now enforced.

Each MCP toolset is wrapped in ApprovalRequiredToolset whose predicate reads the
readOnlyHint the server publishes — the gateway decides what counts as a write,
and the agent keeps no list of tool names in step with it.
output_type=[str, DeferredToolRequests] lets a turn end holding a proposed call
instead of making it; the executor parks the A2A task in input-required and
renders the arguments the model actually sent, not what it said it would do.
The reply is matched against a closed set of words, so "oui mais change le prix"
denies the call and the correction comes back for approval on its own merits —
verified end to end, along with a draft the user approved that was still refused
because msrp sat below buyprice.

Also wired: a ProductDraft validated after approval and before the call leaves;
UsageLimits on every run; FilteredToolset denying sales' write tools inside the
agent as well as at the gateway; PrefixedToolset namespacing the two servers;
dynamic instructions naming the caller; Agent.instrument_all() for a span per
model call and per tool call; and pydantic_ai.retries on inference only, so a
429 from the gateway is waited out rather than failing mid-reasoning.

Inference gets its own httpx client: validate_response raises on any 4xx, and
applying that to the shared client would turn a token-exchange refusal into an
exception carrying no body — which is where Keycloak says what was wrong. Its
transport is built explicitly, because a transport passed to the client makes
verify= inert and the lab CA would otherwise go undeclared.

Tests grew from 7 to 31, covering what approval decides and what a draft must
satisfy. evals/catalogue_evals.py adds a behavioural suite, kept out of pytest
because every case spends real tokens; its first case checks that a count comes
from the tool rather than from what the model remembers of a public dataset.

CLAUDE.md corrected: instrument=True is not an Agent parameter in 2.35.3,
pydantic_ai.retries needs tenacity, TemplateStr needs pydantic-handlebars, and
approval does not remove the in-memory history — resuming still needs the
messages, so replicas: 1 stands until they live in the A2A task store.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-29 09:15:33 +02:00

9.0 KiB

Raw Blame History

Catalogue — product management agent

Framework: PydanticAI + a2a-sdk · Mode: conversational · Client id: classic-models-agent-catalogue

Serves the product manager: create and update products and product lines, set prices, decide what to discontinue.

Scopes and MCP servers


MCP	catalog (`mcp-catalog-audience`), sales (`mcp-sales-audience`)
Scopes	`catalog:read`, `catalog:write`, `sales:read`

sales:read is there to answer "does this product actually sell?" before a pricing or discontinuation decision. Read only — this agent never touches orders.

A2A

Calls — Approvisionnement.check_availability, to know remaining stock and open orders before discontinuing a reference.

Exposes — one read-only skill, describe_catalog: what exists in the catalog, how it is priced and positioned, and whether it sells. Skills are decision-shaped, never a generic query — add one when another agent has a specific question, not in advance.

Why PydanticAI here

Not arbitrary. This is the agent that writes structured data under constraints — prices, quantities, references, product line membership. Typed validation of the model's output earns its place here in a way it would not on a purely conversational agent.

That argument is not yet cashed in. agent.py builds Agent(model, instructions=..., toolsets=...) with no output_type, no output validator and no local tool, so the output is a free str. On the code as it stands, PydanticAI is a bare MCP client and nothing more. Either wire the validation (see below) or stop claiming it as the reason for the framework.

What is wired, and what it cost

All of the below is in the code as of 2026-08-29. Verified against the installed pydantic-ai-slim 2.35.3 — three of these do add a dependency, contrary to what an earlier revision of this file claimed:

pydantic_ai.retries ships in the package but imports tenacity, which the slim extras do not pull. Added.
pydantic-evals is a separate distribution. Added as a dev dependency.
TemplateStr needs pydantic-handlebars (pydantic-ai-slim[spec]). Not added — a template engine earns nothing against a single interpolation, so the dynamic instructions are a plain function, which is the idiomatic API and needs nothing.

Approval as a mechanism, not a prompt rule. Done, and it is the change that matters. Each MCP toolset is wrapped in ApprovalRequiredToolset, whose predicate reads the readOnlyHint the server publishes — so the gateway decides what counts as a write and the agent keeps no list of tool names in step with it. output_type=[str, DeferredToolRequests] lets a turn end holding a proposed call instead of making it; the executor parks the task with TaskUpdater.requires_input() and renders the arguments the model actually proposed, not what it said it would do. The reply is read deterministically against a closed set of words — a model must not be what decides whether the user said yes, and "oui mais change le prix" is not consent.

One claim in the earlier revision does not hold: this does not let the in-memory history go. Resuming needs message_history, so the pending state still lives in the process and replicas: 1 still stands. Moving it into the A2A task store is what would lift that, and it is not done.
A validated draft on writes. approval.py holds ProductDraft — price above zero, stock within range, msrp above buyprice, no empty strings, no extra fields. It runs after the user approves and before the call leaves, so a record the user waved through is still refused if the model built it wrong. Note this is argument validation, not output_type: output_type describes a run's result, and the thing worth constraining here is what goes out, not what comes back.
Instrumentation. Agent.instrument_all() at start-up, in __main__.py — not instrument=True on Agent(...), which does not exist in 2.35.3; the parameters are capabilities, defer_model_check, deps_type, description, end_strategy, instructions, max_concurrency, metadata, model, model_settings, name, output_type, retries, system_prompt, tool_timeout, tools, toolsets, validation_context. OpenTelemetry and logfire_api are already present transitively. One span per model call and per tool call: the MCP server records which tool ran, this records why.
UsageLimits(tool_calls_limit=...) on every run, from Settings.max_tool_calls (20).
Toolset shaping. FilteredToolset denies sales' write tools inside the agent as well as at the gateway — an admin reaching this agent would otherwise be offered create-order. PrefixedToolset namespaces each server, so a future collision between two domains is impossible rather than silent.
Dynamic instructions. A caller_instructions(ctx) function appended per run, naming the identity the agent is acting under. Deps-typed (CallerDeps), no template engine — see above.
pydantic_ai.retries on inference only. wait_retry_after honours the gateway's own 429 AI token rate limit exceeded, so a turn survives the limit instead of failing mid-reasoning. It gets a separate httpx client: validate_response raises on any 4xx, and applying that to the shared client would turn a token-exchange refusal into an exception carrying no body — which is where Keycloak says what was wrong.
durable_exec — not done, on the reasoning in the earlier revision: heavy, and the right fix for single-replica history is the A2A task store, not a workflow engine.

pydantic-evals, now a dev dependency, with a suite in evals/catalogue_evals.py. It is not part of pytest: every case spends real tokens through the gateway and needs a live realm.

Its cases are the failures actually seen, not coverage. The first one is the important one: an agent asked for a product count once answered the true figure while its tool had returned a page holding a different one. Right answer, green trace, ungrounded number — and only a check against the data tells the two apart.

The unit tests grew to 31 alongside, covering what approval decides and what a draft must satisfy. Every rule they check used to live in the system prompt, which is to say it used to be a request.

Keep the split straight while doing any of this: none of it is authorisation. Who may write is decided by the MCP server against the end user's own roles, at discovery time. Everything above makes the agent predictable, not privileged.

a2a-sdk, not FastA2A

An earlier revision of this file said FastA2A, on the grounds that Agent.to_a2a() was going away in PydanticAI v2 — which has since happened, so that argument no longer separates the two. The code uses the official a2a-sdk (1.1.x): this estate exists to prove A2A interoperability across five runtimes, and the reference implementation is the safer thing for four other stacks to interoperate with.

It is framework-agnostic too, so it stays the natural basis for the A2A adapter in front of the n8n agents — see classic-models-agent-flows.

Before writing code

Read classic-models-agent-platform/CLAUDE.md first. It holds the identity model, the MCP and LLM endpoints, the token-exchange recipe and the known blockers. This file only covers what is specific to this agent.

Non-negotiables

Inference goes through the estate's LLM endpoint, https://llm-dev.sttlab.local/llm/v1/chat/completions. Calling a model provider directly bypasses the rate limits and prompt guards.
One MCP client per conversation turn, built with freshly exchanged tokens. Do not hold a long-lived MCP connection with a fixed token.
A2A calls return answers, not data. Expose narrow decision-shaped skills, never a generic query.
Deliverables in English.
Never name the middleware. The agent talks to URLs that honour a contract: an MCP endpoint, an OIDC token endpoint, an OpenAI-shaped LLM endpoint. What serves them is replaceable and the agent must not know or care. Write "the MCP server", "the LLM endpoint", "the gateway" — never a product name, in code, comments, commits or docs.

Running it against the lab from macOS

macOS resolves every .local name through mDNS before falling back to /etc/hosts, and that costs a flat ~5.0s per hostname — dscacheutil -q host -a name catalog-dev.sttlab.local shows it, and so does curl -w %{time_namelookup}. httpx's default connect timeout is 5s, so the very first request to each host lands on the boundary and fails intermittently. fastmcp reports that as Failed to initialize server session and swallows the cause, which is why it read as an MCP protocol problem for a long time. curl --resolve hides it completely — do not use --resolve to "prove" DNS is fine.

Settings.http_timeout (60s) is the fix, applied to the shared httpx2 client, to the client fastmcp builds, and to MCPToolset(init_timeout=...). In the cluster these names resolve instantly and the timeout never binds.

9.0 KiB Raw Blame History