fix(llm): double the per-client token budget to 100k/min
A tool-calling agent re-sends the whole tool result on every iteration of its loop, so a single question against a tool returning a full collection burns the old budget in three turns and answers 429 in the middle of reasoning. Pagination on those tools is the real fix; this buys room until then and stays well inside the 500k global ceiling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -40,7 +40,13 @@ spec:
|
||||
VECTORDB_HOST: llm-dev-vectordb
|
||||
VECTORDB_PORT: "6379"
|
||||
RATE_LIMIT_GLOBAL_TOKENS_PER_MINUTE: "500000"
|
||||
RATE_LIMIT_PER_CLIENT_TOKENS_PER_MINUTE: "50000"
|
||||
# Doubled from 50000. A tool-calling agent re-sends the whole tool
|
||||
# result on every iteration of its loop, so one question against a
|
||||
# tool that returns a full collection (list-products is ~23 KB) can
|
||||
# burn the old budget in three turns and answer 429 mid-reasoning.
|
||||
# Pagination on those tools is the real fix; this buys room until
|
||||
# then, and stays well inside the global ceiling.
|
||||
RATE_LIMIT_PER_CLIENT_TOKENS_PER_MINUTE: "100000"
|
||||
# Looked up after consumer.yaml's llm-anonymous KongConsumer first
|
||||
# reconciled — config.anonymous needs the real Consumer ID, not the
|
||||
# username (Konnect stores it as a literal string otherwise, silently
|
||||
|
||||
Reference in New Issue
Block a user