Risk Management for Autonomous AI Systems: Budget Caps, Circuit Breakers, and Kill Switches
Autonomous systems can fail silently, spend money without bounds, or cascade errors through dependent components. Without explicit risk management, a single malfunctioning agent loop can cost hundreds of dollars before anyone notices. A trading algorithm can lose the account. A content generator can publish embarrassing output.
Risk management for autonomous AI is not optional. Here are the three layers of defense that every system needs.
Layer 1: Budget Caps (Non-Negotiable)
Hard spending limits that cannot be overridden by any agent or subsystem. The budget check runs BEFORE every operation, not after.
- Monthly system-wide cap (e.g., $500)
- Per-organism monthly cap (e.g., $100)
- Per-generation LLM API cap (e.g., $50 per 2-week cycle)
- Per-trade maximum (e.g., $50 for trading systems)
- Alert at 80%, halt at 100%
Writing budget limits in documentation is useless. They must be enforced in code with automatic halt.
Layer 2: Circuit Breakers
Automatic disconnection of failing components. Three states: CLOSED (normal), OPEN (all blocked), HALF-OPEN (one test request allowed).
Circuit breakers operate at three scales:
- Component-level: Individual API calls, model invocations
- Organism-level: Entire income-generating products
- System-level: Global halt when total error rates exceed bounds
For trading systems, a two-tier breaker: at 50% of daily loss limit, cut position size in half. At 100%, close all positions and halt.
Layer 3: Kill Switches
Immediate stop for any component. Cannot be overridden. Every organism needs one. Always log: the reason, the state at time of halt, and the steps needed to investigate.
Defense-in-Depth Architecture
- Pre-execution: Budget check, circuit breaker check, kill switch check, dependency health check
- During execution: Real-time cost tracking, quality gates on outputs, timeout enforcement
- Post-execution: Anomaly detection, budget update, audit logging
- Periodic review: Risk register review, budget vs actuals, circuit breaker history analysis
The layers are independent. If one fails, the others still protect.
Risk Management (Pattern 13) and Circuit Breakers (Pattern 14) are detailed with full TypeScript implementations in the Protocol Playbook.