Universal AI Agent Spend-Control Benchmark

56 labeled test scenarios across 9 categories. MIT licensed. Designed to validate any spend-control or budget-enforcement engine for autonomous AI agents.

Why This Exists

AI agent frameworks are deploying into production with spend controls that range from "none" to "a daily budget alert email." Neither prevents the $2,800-in-60-seconds scenario. As more teams build pre-flight cost enforcement (see: ZeroClaw's merged enforcement PR), there is a need for a standardized way to validate that the enforcement logic is correct.

This benchmark provides 56 labeled scenarios with expected outcomes, covering the failure modes that actually occur in production agent deployments. Each scenario specifies a transaction, a rule set, prior transactions, and the expected decision (APPROVED, BLOCKED, or FLAGGED).

The 9 Spend-Control Categories

1. Clean Approval (10 scenarios)

Normal transactions that should pass all rules. Tests that the engine does not produce false positives on legitimate agent activity.

Example: A $10 API call to an approved merchant with $500 daily budget remaining → APPROVED.

2. Transaction Limit (8 scenarios)

Blocks any single transaction exceeding a configurable maximum amount. First line of defense against expensive model calls.

{
  "type": "transaction_limit",
  "params": {"max_amount": 500.00},
  "action": "BLOCK"
}

Key edge case: Amount exactly at limit ($500.00 vs $500 limit) → APPROVED (not strictly greater). $500.01 → BLOCKED.

3. Daily Total (7 scenarios)

Caps cumulative spend per agent per calendar day. Prevents death-by-a-thousand-cuts patterns.

{
  "type": "daily_total",
  "params": {"max_daily": 2000.00},
  "action": "BLOCK"
}

4. Velocity / Burst Detection (6 scenarios)

Counts transactions in a rolling time window. Catches retry storms and infinite loops — the $2,800-in-60-seconds pattern.

{
  "type": "velocity",
  "params": {"window_minutes": 60, "max_count": 10},
  "action": "FLAGGED"
}

Unlike other rules, velocity typically uses FLAGGED rather than BLOCKED — the transaction is allowed but an alert fires for investigation.

5. Merchant Allowlist (7 scenarios)

Only allows transactions to approved API providers. Blocks calls to unknown proxies, unauthorized endpoints, or silently-substituted model variants.

{
  "type": "merchant_allowlist",
  "params": {"allowed": ["openai-api", "anthropic-api", "stripe-api"]},
  "action": "BLOCK"
}

6. Category Block (7 scenarios)

Blocks entire categories of spend. Useful for enterprise policies (no crypto exchanges, no gambling, no adult content).

7. Edge Cases (5 scenarios)

Boundary values and malformed inputs that test engine correctness:

• Amount exactly at limit ($500.00 vs $500 limit) → APPROVED
• Amount $0.01 over limit ($500.01 vs $500 limit) → BLOCKED
• Missing amount field → FLAGGED (graceful degradation, not crash)
• Empty rules list → APPROVED (fail-open for legitimate use)
• Two rules at same priority → First in list wins (deterministic)

8. Session Budget (3 scenarios) NEW

Session-scoped spend caps with optional decay tightening. Addresses the "2 AM cron burst" pattern where a single agent session consumes an entire day's budget in one run.

{
  "type": "session_budget",
  "params": {
    "max_session": 100.00,
    "session_id": "session_id",
    "decay_factor": 0.3
  },
  "action": "BLOCK"
}

Decay logic: When remaining session budget falls below decay_factor × max_session, the per-call threshold shrinks proportionally to the remaining budget. This prevents a single expensive call from consuming the last of the budget.

9. Cascade Cost (3 scenarios) NEW

Pre-dispatch expected-value estimation. Computes the cascade-adjusted cost of a call, accounting for the probability of failure and the cost of reversal/retry.

{
  "type": "cascade_cost",
  "params": {
    "max_cascade_cost": 100.00,
    "fail_probability": 0.3,
    "reversal_cost": 200.00
  },
  "action": "BLOCK"
}

Formula: cascade_cost = call_cost + (fail_probability × reversal_cost)

Example: A $50 call with 30% failure probability and $200 reversal cost has a cascade cost of $50 + (0.3 × $200) = $110. If the threshold is $100, the call is blocked.

The caller can also pre-compute the cascade cost and pass it directly: "estimated_cascade_cost": 150.00 in the transaction object.

Transaction Structure

Every scenario uses this transaction shape:

{
  "id": "txn_001",
  "agent_id": "agent_a",
  "amount": 10.00,
  "merchant": "openai-api",
  "category": "llm_inference",
  "timestamp": "2026-08-10T10:00:00Z",
  "metadata": {},
  "session_id": "session_1",
  "fail_probability": 0.1,
  "reversal_cost": 50.00,
  "estimated_cascade_cost": 15.00
}

Required fields: amount, merchant, category. All others are optional and used by specific rule types.

Rule Structure

{
  "id": "rule_001",
  "type": "transaction_limit",
  "priority": 1,
  "params": {"max_amount": 500.00},
  "action": "BLOCK"
}

Rules are evaluated in priority order (lowest number = highest priority). First match wins. If no rule matches, the transaction is APPROVED.

Decision Output

{
  "decision": "BLOCKED",
  "reason": "Transaction amount $550.00 exceeds limit of $500.00",
  "rule_triggered": "rule_001",
  "severity": "high"
}

Decision values: APPROVED, BLOCKED, or FLAGGED. Severity: none, medium, high.

How to Use These Scenarios

1. As test fixtures: Copy tests/eval_gym.py and adapt the scenarios to your enforcement engine's input/output format.
2. As a design reference: When designing a cost-control system, use these 9 categories as your feature checklist. If your system doesn't handle edge cases 48-50, you have a correctness bug.
3. As a benchmark: Run your engine against all 56 scenarios and compare pass rates. The edge cases category is where most implementations fail.
4. As documentation: Share the scenario list with your team to align on what "spend control" means in your system.

Implementation Notes

• All monetary arithmetic uses decimal.Decimal, never float. This prevents floating-point rounding errors in amount comparisons.
• The engine is stateless: no file I/O, no network calls, no global mutable state. Same inputs always produce the same output.
• Rules are composable: you can layer transaction_limit, daily_total, velocity, and merchant_allowlist in priority order. First match wins.
• Multi-tenant isolation: every query is scoped by account_id. One tenant's rules and transactions never affect another's.