Stéphane Tailland
1a13743078
The agent asked for confirmation because the system prompt told it to, and read the answer itself. Both halves are now enforced. Each MCP toolset is wrapped in ApprovalRequiredToolset whose predicate reads the readOnlyHint the server publishes — the gateway decides what counts as a write, and the agent keeps no list of tool names in step with it. output_type=[str, DeferredToolRequests] lets a turn end holding a proposed call instead of making it; the executor parks the A2A task in input-required and renders the arguments the model actually sent, not what it said it would do. The reply is matched against a closed set of words, so "oui mais change le prix" denies the call and the correction comes back for approval on its own merits — verified end to end, along with a draft the user approved that was still refused because msrp sat below buyprice. Also wired: a ProductDraft validated after approval and before the call leaves; UsageLimits on every run; FilteredToolset denying sales' write tools inside the agent as well as at the gateway; PrefixedToolset namespacing the two servers; dynamic instructions naming the caller; Agent.instrument_all() for a span per model call and per tool call; and pydantic_ai.retries on inference only, so a 429 from the gateway is waited out rather than failing mid-reasoning. Inference gets its own httpx client: validate_response raises on any 4xx, and applying that to the shared client would turn a token-exchange refusal into an exception carrying no body — which is where Keycloak says what was wrong. Its transport is built explicitly, because a transport passed to the client makes verify= inert and the lab CA would otherwise go undeclared. Tests grew from 7 to 31, covering what approval decides and what a draft must satisfy. evals/catalogue_evals.py adds a behavioural suite, kept out of pytest because every case spends real tokens; its first case checks that a count comes from the tool rather than from what the model remembers of a public dataset. CLAUDE.md corrected: instrument=True is not an Agent parameter in 2.35.3, pydantic_ai.retries needs tenacity, TemplateStr needs pydantic-handlebars, and approval does not remove the in-memory history — resuming still needs the messages, so replicas: 1 stands until they live in the A2A task store. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
171 lines
9.0 KiB
Markdown
171 lines
9.0 KiB
Markdown
# Catalogue — product management agent
|
|
|
|
**Framework:** PydanticAI + `a2a-sdk` · **Mode:** conversational · **Client id:** `classic-models-agent-catalogue`
|
|
|
|
Serves the product manager: create and update products and product lines, set
|
|
prices, decide what to discontinue.
|
|
|
|
## Scopes and MCP servers
|
|
|
|
| | |
|
|
|---|---|
|
|
| MCP | catalog (`mcp-catalog-audience`), sales (`mcp-sales-audience`) |
|
|
| Scopes | `catalog:read`, `catalog:write`, `sales:read` |
|
|
|
|
`sales:read` is there to answer "does this product actually sell?" before a
|
|
pricing or discontinuation decision. Read only — this agent never touches orders.
|
|
|
|
## A2A
|
|
|
|
**Calls** — `Approvisionnement.check_availability`, to know remaining stock and
|
|
open orders before discontinuing a reference.
|
|
|
|
**Exposes** — one read-only skill, `describe_catalog`: what exists in the
|
|
catalog, how it is priced and positioned, and whether it sells. Skills are
|
|
decision-shaped, never a generic `query` — add one when another agent has a
|
|
specific question, not in advance.
|
|
|
|
## Why PydanticAI here
|
|
|
|
Not arbitrary. This is the agent that **writes structured data under
|
|
constraints** — prices, quantities, references, product line membership. Typed
|
|
validation of the model's output earns its place here in a way it would not on a
|
|
purely conversational agent.
|
|
|
|
**That argument is not yet cashed in.** `agent.py` builds `Agent(model,
|
|
instructions=..., toolsets=...)` with no `output_type`, no output validator and
|
|
no local tool, so the output is a free `str`. On the code as it stands,
|
|
PydanticAI is a bare MCP client and nothing more. Either wire the validation
|
|
(see below) or stop claiming it as the reason for the framework.
|
|
|
|
## What is wired, and what it cost
|
|
|
|
All of the below is in the code as of 2026-08-29. Verified against the installed
|
|
`pydantic-ai-slim` 2.35.3 — **three of these do add a dependency**, contrary to
|
|
what an earlier revision of this file claimed:
|
|
|
|
- `pydantic_ai.retries` ships in the package but imports `tenacity`, which the
|
|
slim extras do not pull. Added.
|
|
- `pydantic-evals` is a separate distribution. Added as a dev dependency.
|
|
- `TemplateStr` needs `pydantic-handlebars` (`pydantic-ai-slim[spec]`). **Not
|
|
added** — a template engine earns nothing against a single interpolation, so
|
|
the dynamic instructions are a plain function, which is the idiomatic API and
|
|
needs nothing.
|
|
|
|
1. **Approval as a mechanism, not a prompt rule.** Done, and it is the change
|
|
that matters. Each MCP toolset is wrapped in `ApprovalRequiredToolset`, whose
|
|
predicate reads the **`readOnlyHint` the server publishes** — so the gateway
|
|
decides what counts as a write and the agent keeps no list of tool names in
|
|
step with it. `output_type=[str, DeferredToolRequests]` lets a turn end
|
|
holding a proposed call instead of making it; the executor parks the task
|
|
with `TaskUpdater.requires_input()` and renders the arguments *the model
|
|
actually proposed*, not what it said it would do. The reply is read
|
|
deterministically against a closed set of words — a model must not be what
|
|
decides whether the user said yes, and "oui mais change le prix" is not
|
|
consent.
|
|
|
|
One claim in the earlier revision does not hold: this does **not** let the
|
|
in-memory history go. Resuming needs `message_history`, so the pending state
|
|
still lives in the process and `replicas: 1` still stands. Moving it into the
|
|
A2A task store is what would lift that, and it is not done.
|
|
2. **A validated draft on writes.** `approval.py` holds `ProductDraft` —
|
|
price above zero, stock within range, msrp above buyprice, no empty strings,
|
|
no extra fields. It runs *after* the user approves and *before* the call
|
|
leaves, so a record the user waved through is still refused if the model
|
|
built it wrong. Note this is argument validation, not `output_type`:
|
|
`output_type` describes a run's result, and the thing worth constraining
|
|
here is what goes out, not what comes back.
|
|
3. **Instrumentation.** `Agent.instrument_all()` at start-up, in
|
|
`__main__.py` — **not** `instrument=True` on `Agent(...)`, which does not
|
|
exist in 2.35.3; the parameters are `capabilities, defer_model_check,
|
|
deps_type, description, end_strategy, instructions, max_concurrency,
|
|
metadata, model, model_settings, name, output_type, retries, system_prompt,
|
|
tool_timeout, tools, toolsets, validation_context`. OpenTelemetry and
|
|
`logfire_api` are already present transitively. One span per model call and
|
|
per tool call: the MCP server records *which* tool ran, this records why.
|
|
4. **`UsageLimits(tool_calls_limit=...)`** on every run, from
|
|
`Settings.max_tool_calls` (20).
|
|
5. **Toolset shaping.** `FilteredToolset` denies sales' write tools inside the
|
|
agent as well as at the gateway — an admin reaching this agent would
|
|
otherwise be offered `create-order`. `PrefixedToolset` namespaces each
|
|
server, so a future collision between two domains is impossible rather than
|
|
silent.
|
|
6. **Dynamic instructions.** A `caller_instructions(ctx)` function appended
|
|
per run, naming the identity the agent is acting under. Deps-typed
|
|
(`CallerDeps`), no template engine — see above.
|
|
7. **`pydantic_ai.retries`** on inference only. `wait_retry_after` honours
|
|
the gateway's own `429 AI token rate limit exceeded`, so a turn survives the
|
|
limit instead of failing mid-reasoning. It gets a **separate httpx client**:
|
|
`validate_response` raises on any 4xx, and applying that to the shared client
|
|
would turn a token-exchange refusal into an exception carrying no body —
|
|
which is where Keycloak says what was wrong.
|
|
8. **`durable_exec`** — not done, on the reasoning in the earlier revision:
|
|
heavy, and the right fix for single-replica history is the A2A task store,
|
|
not a workflow engine.
|
|
|
|
**`pydantic-evals`**, now a dev dependency, with a suite in
|
|
`evals/catalogue_evals.py`. It is not part of `pytest`: every case spends real
|
|
tokens through the gateway and needs a live realm.
|
|
|
|
Its cases are the failures actually seen, not coverage. The first one is the
|
|
important one: an agent asked for a product count once answered the true figure
|
|
while its tool had returned a page holding a different one. Right answer, green
|
|
trace, ungrounded number — and only a check against the data tells the two
|
|
apart.
|
|
|
|
The unit tests grew to 31 alongside, covering what approval decides and what a
|
|
draft must satisfy. Every rule they check used to live in the system prompt,
|
|
which is to say it used to be a request.
|
|
|
|
Keep the split straight while doing any of this: none of it is authorisation.
|
|
Who may write is decided by the MCP server against the end user's own roles, at
|
|
discovery time. Everything above makes the agent predictable, not privileged.
|
|
|
|
## a2a-sdk, not FastA2A
|
|
|
|
An earlier revision of this file said FastA2A, on the grounds that
|
|
`Agent.to_a2a()` was going away in PydanticAI v2 — which has since happened, so
|
|
that argument no longer separates the two. The code uses the official
|
|
`a2a-sdk` (1.1.x): this estate exists to prove A2A interoperability across five
|
|
runtimes, and the reference implementation is the safer thing for four other
|
|
stacks to interoperate with.
|
|
|
|
It is framework-agnostic too, so it stays the natural basis for the A2A adapter
|
|
in front of the n8n agents — see `classic-models-agent-flows`.
|
|
|
|
## Before writing code
|
|
|
|
Read `classic-models-agent-platform/CLAUDE.md` first. It holds the identity
|
|
model, the MCP and LLM endpoints, the token-exchange recipe and the known
|
|
blockers. This file only covers what is specific to this agent.
|
|
|
|
## Non-negotiables
|
|
|
|
- **Inference goes through the estate's LLM endpoint**, `https://llm-dev.sttlab.local/llm/v1/chat/completions`.
|
|
Calling a model provider directly bypasses the rate limits and prompt guards.
|
|
- **One MCP client per conversation turn**, built with freshly exchanged tokens.
|
|
Do not hold a long-lived MCP connection with a fixed token.
|
|
- **A2A calls return answers, not data.** Expose narrow decision-shaped skills,
|
|
never a generic `query`.
|
|
- Deliverables in English.
|
|
- **Never name the middleware.** The agent talks to URLs that honour a contract:
|
|
an MCP endpoint, an OIDC token endpoint, an OpenAI-shaped LLM endpoint. What
|
|
serves them is replaceable and the agent must not know or care. Write "the
|
|
MCP server", "the LLM endpoint", "the gateway" — never a product name, in
|
|
code, comments, commits or docs.
|
|
|
|
## Running it against the lab from macOS
|
|
|
|
macOS resolves every `.local` name through mDNS before falling back to
|
|
`/etc/hosts`, and that costs a **flat ~5.0s per hostname** — `dscacheutil -q host
|
|
-a name catalog-dev.sttlab.local` shows it, and so does `curl -w
|
|
%{time_namelookup}`. httpx's default connect timeout is 5s, so the very first
|
|
request to each host lands on the boundary and fails intermittently. fastmcp
|
|
reports that as `Failed to initialize server session` and swallows the cause,
|
|
which is why it read as an MCP protocol problem for a long time. `curl --resolve`
|
|
hides it completely — do not use `--resolve` to "prove" DNS is fine.
|
|
|
|
`Settings.http_timeout` (60s) is the fix, applied to the shared httpx2 client,
|
|
to the client fastmcp builds, and to `MCPToolset(init_timeout=...)`. In the
|
|
cluster these names resolve instantly and the timeout never binds.
|