The real cost of running AI agents in production

The prototype worked beautifully. A single agent, a handful of tools, a well-crafted prompt. Inference cost: a few cents per request. The team estimated production costs from that number, multiplied by expected volume, and approved the budget.

Six months later, the agent is in production and the monthly bill is 10x the estimate. Nobody made a mistake. The estimate was accurate for the prototype. It was wrong for production because production requires everything the prototype did not: monitoring, evaluation, retries, escalation, security filtering, and the operational overhead of keeping a non-deterministic system within acceptable bounds.

That operational overhead has a name: stochastic tax.

This is not a new problem in a new costume. Sculley et al. (2015) showed that in production ML systems, the actual machine learning code is a tiny fraction of the total system — the surrounding infrastructure for data collection, feature extraction, monitoring, serving, and configuration management dwarfs it. That pattern carried over to agentic systems. The agentic technical debt framework (forthcoming in Communications of the ACM, October 2026) extends the finding: inference is the part you can see on a bill, but the operational overhead of keeping a non-deterministic system reliable is where the money actually goes.

Key takeaways

• Agent workflows consume roughly 19x more model tokens than simple inference, before accounting for monitoring, evaluation, or human escalation.
• Production costs run 5–15x higher than prototype costs. The multiplier is stochastic tax: the operating overhead of running non-deterministic agents reliably.
• The largest hidden cost is usually human escalation, not LLM tokens — a single escalation can cost more than 100 autonomous transactions.
• Model routing — directing each step to the smallest capable model — can cut inference cost by 50–98% with negligible quality loss.
• Making costs visible per-workflow is the prerequisite to making them manageable.

Where the money goes

The total cost of an agent in production breaks into three layers: inference, infrastructure, and operations. Most teams budget for the first, underestimate the second, and ignore the third.

Layer 1: inference

This is what most teams measure: the cost of LLM API calls. For a simple query-response pattern, this is the dominant cost. For an agentic workflow, it is often less than a third of the total.

Agent workflows are token-intensive. Each step in a multi-step reasoning chain requires a full context window: the system prompt, the conversation history, the tool descriptions, and the results of prior tool calls. A four-step agent workflow with five available tools can easily consume 10,000–20,000 input tokens per request before a single output token is generated.

A recent conceptual framework on token economics for LLM agents (Chen et al., 2026) formalizes this: tokens function simultaneously as factors of production, media of exchange, and units of account in agentic systems. Every reasoning step, tool call, and context injection is a token expenditure — and unlike traditional compute, the cost scales with the complexity of the reasoning, not just the volume of data processed.

The energy dimension compounds this. Luccioni et al. (FAccT 2024) conducted the first systematic comparison of inference energy costs across ML system types and found that multi-purpose generative models are orders of magnitude more expensive in energy consumption than task-specific models, even when controlling for parameter count. An agent workflow that routes every step through a frontier generative model is paying that energy premium on every token.

Layer 2: infrastructure

The compute and storage that surrounds the model:

• Vector stores and retrieval. RAG pipelines add embedding cost, storage cost, and retrieval latency. Every agent request that queries a knowledge base pays this.
• Tool hosting. External tools — APIs, databases, search indices — have their own per-call costs. An agent that makes three tool calls per request triples this line item compared to a non-agentic system.
• Logging and storage. Full traces — inputs, reasoning steps, tool calls, outputs — need to be stored for debugging, evaluation, and compliance. At scale, trace storage becomes a non-trivial cost.
• Compute for evaluation. Golden-set evaluations and automated quality scoring run LLM inference on evaluation prompts. This is inference cost that produces no user-facing output — it exists purely to verify that the agent is behaving correctly.

Layer 3: operations

This is where the prototype-to-production gap lives.

• Monitoring and observability. Real-time dashboards, alerting, anomaly detection — the infrastructure that tells you when something is wrong. This is both tooling cost and the team’s time spent watching.
• Human escalation. The most expensive per-incident cost in the system. A human reviewer costs $20–80 per hour. If 10% of agent transactions escalate and each escalation takes 5 minutes, the human cost per transaction is $1–7 for those 10% — far more than the LLM inference cost for the 90% that ran autonomously. Graduated autonomy manages this by routing only genuinely ambiguous or high-risk tasks to humans.
• Retries. When an agent fails mid-task, the retry consumes fresh tokens and fresh tool calls. Well-designed loop patterns cap retry costs with circuit breakers and bounded execution. Poorly designed ones let retries cascade.
• Revalidation. Every model update, prompt change, or tool modification triggers a round of golden-set evaluation. This is the “change tax” — the cost of confirming that nothing regressed.
• Security and compliance. Input/output filtering, prompt injection defenses, PII detection, audit logging. These scale with transaction volume and action scope.

As we argued in Harvard Business Review, deploying an AI agent is a workforce decision. The cost analogy holds: just as an employee’s total cost of employment far exceeds their salary — benefits, training, management overhead, office space, compliance — an agent’s total cost of operation far exceeds its inference bill. Teams that budget only for inference are making the same mistake as a company that budgets only for base salary.

The prototype-to-production multiplier

Why is the multiplier 5–15x rather than 2x?

Because prototype costs only include Layer 1 (inference for the happy path). Production costs include all three layers, plus the variance costs: retries on the unhappy path, escalations on edge cases, monitoring of all paths, and evaluation of every deployment.

The multiplier also depends on the complexity of the workflow:

Workflow type	Typical multiplier	Why
Single-agent, narrow scope	3–5x	Low escalation rate, few tools, simple monitoring
Multi-agent, sequential chain	8–12x	Latency compounding, handoff retries, shared-state debugging
Multi-agent, complex graph	10–15x	All of the above plus orchestration overhead

These are operating cost multipliers, not infrastructure cost multipliers. The cloud compute bill may only double. The total operating cost — including human time, evaluation cycles, and opportunity cost of incidents — is what multiplies by 10.

Making costs visible

You cannot manage what you cannot see. Most teams track aggregate LLM spend. Few track cost per workflow, per step, or per transaction. The gap between those two views is where stochastic tax hides.

Per-workflow cost tracking

Break your total agent spend into individual workflows. Each workflow should have its own cost dashboard showing:

• Token cost (input + output + reasoning) per transaction
• Tool-call cost per transaction
• Escalation cost per transaction (human time x hourly rate)
• Retry cost per transaction (additional tokens + tool calls on failed attempts)
• Evaluation cost per deployment (amortized across transactions)

The per-transaction total is your stochastic tax for that workflow. Track it over time. A rising trend without a corresponding increase in task complexity is a signal that agentic technical debt is accumulating — memory bloat that silently inflates context size is a common culprit.

Per-step cost attribution

Within a workflow, identify which steps consume the most resources. This is often surprising. The LLM reasoning step that seems expensive may cost less than the tool call that queries a paid external API, or the human review step that triggers on 15% of transactions.

Per-step attribution tells you where optimization effort has the highest return. Replacing an expensive model call with a smaller model saves money only if that step is a significant share of the total cost. If the dominant cost is human escalation, model optimization moves the wrong needle.

Six strategies to reduce operating cost

The research on LLM cost optimization has matured rapidly. These strategies are ordered from quickest wins to structural changes, with the academic evidence behind each.

1. Right-size models per step

Use a capable model for reasoning and judgment. Use a fast model for classification, routing, extraction, and formatting. The cost difference can be 10–50x per token, and the quality difference on routine tasks is often negligible. A model gateway makes the routing transparent — the application sends a request with metadata, and the gateway selects the model.

FrugalGPT (Chen et al., TMLR 2024) formalized this as LLM cascading: learning which model combination to use per query can reduce costs by up to 98% while matching or exceeding the best individual model’s performance. The key insight is that most queries do not need the most capable model — a smaller model handles them correctly, and the expensive model is reserved for the cases where the smaller one is uncertain.

Two recent systems make this practical at scale. RouteLLM (Ong et al., ICLR 2025) trains lightweight router models on human preference data to dynamically select between a stronger and weaker LLM, achieving over 2x cost reduction while preserving response quality. AutoMix (Aggarwal et al., NeurIPS 2024) uses a POMDP-based router with self-verification to escalate from a smaller model to a larger one only when needed, cutting computational cost by over 50% without fine-tuned verifiers.

2. Move deterministic work out of the LLM

Format validation, policy rule checks, data transformation, routing logic — every task that can be expressed as code should be code. Deterministic checks cost microseconds and zero tokens. An LLM call that enforces a format constraint costs seconds and thousands of tokens. Tool schema contracts formalize this boundary.

3. Reduce escalation rate

Escalation is often the largest cost component. Two approaches:

• Improve the agent. If escalations cluster around specific input types, the agent may need better instructions, better tools, or more context for those cases. Golden-set evaluation identifies which input types escalate most often.
• Adjust the autonomy tier. If the agent escalates tasks it could handle, the graduated autonomy thresholds may be too conservative. Promoting stable tasks from yellow to green eliminates escalation cost for those tasks — a single promotion on a 10,000-transaction/month workflow saves $2,000–8,000/month in human review.

4. Cache aggressively

Many agent requests share common sub-tasks: retrieving the same knowledge base articles, looking up the same customer data, generating the same boilerplate. Caching tool-call results and intermediate reasoning reduces token cost and latency for repeated patterns. Prompt caching — reusing the cached prefix of system prompts and tool descriptions across requests — is now supported by major providers and can cut input token costs significantly on long system prompts.

5. Parallelize sequential chains

Sequential agent chains compound latency at each step. If two steps do not depend on each other, running them in parallel cuts wall-clock time and reduces timeout-driven retries. This is a structural change to the workflow graph, not an optimization — and it is often the single highest-impact cost reduction.

BudgetMLAgent (Gandhi et al., AIMLSystems 2024) demonstrated the combined effect of these strategies in a multi-agent setting: using LLM profiling, cascaded model selection, and an “ask-the-expert” fallback pattern, they achieved a 94.2% cost reduction (from $0.93 to $0.054 per task) while actually improving success rates compared to a GPT-4-only baseline. The lesson: cost optimization and quality improvement are not always in tension.

6. Budget stochastic tax explicitly

Stop treating operational overhead as a surprise. Set a stochastic tax budget per workflow, the same way you would set an SLA for latency or availability. If the actual tax exceeds the budget, it triggers a review — is debt amplifying the tax, or is the task inherently expensive?

Strategy	Typical cost reduction	Where it applies
Right-size models per step	50–98% of inference cost	Multi-step workflows with mixed-complexity steps
Move deterministic work to code	100% of tokens for moved constraints	Policy rules, format validation, routing logic
Reduce escalation rate	$2,000–8,000/month per promoted action	Workflows with stable, over-escalated tasks
Cache aggressively	20–40% of repeated token cost	Workflows with common sub-tasks or shared context
Parallelize sequential chains	25–40% of latency-driven retry cost	Sequential chains with independent steps
Budget stochastic tax	Prevents unbounded cost growth	All production workflows

The ROI question

The cost of running AI agents is only half the equation. The other half is the value they create. An agent that costs $8 per transaction in stochastic tax but replaces a $50 manual process is a 6x return. An agent that costs $8 per transaction and produces results a human needs to redo half the time has negative ROI.

The three-stage agentic AI ROI model provides a framework for this calculation: cost savings (replacing manual work), revenue generation (enabling new capabilities), and new possibilities (workflows that were not feasible manually).

Stochastic tax is the denominator in all three. Making it visible, measurable, and manageable is the prerequisite to proving that your agents are worth what they cost.

What to do next

If your agent cost tracking stops at “monthly LLM spend,” start by decomposing one workflow into its three layers and computing the per-transaction total. The gap between what you thought it cost and what it actually costs is almost always surprising — and it is always actionable.

For the full governance framework — including the five debt mechanisms and the five controls that bring costs under control — see Agentic technical debt: a governance framework for AI agents.