The orchestration jungle: when chaining AI agents breaks

You started with one agent. It worked. So you added a second to handle the next step, and a third for the edge cases. Each agent passes its output to the next. The chain grows. Six months later, modifying any single agent requires revalidating the entire workflow, end-to-end latency has tripled, and your operating costs are climbing with no corresponding increase in capability.

Welcome to the orchestration jungle.

The term comes from the agentic technical debt framework (forthcoming in Communications of the ACM, October 2026), which defines it as the agentic equivalent of the pipeline jungle — an anti-pattern first identified by Sculley et al. (2015) in ML data pipelines. Where pipeline jungles are tangles of data-preparation logic, orchestration jungles are tangles of agent handoffs, shared state, and retry logic that become so intertwined that no single agent can be changed in isolation.

The problem is not new. Distributed systems hit the same wall decades ago with microservice choreography. What is new is the combination of non-deterministic execution and natural-language interfaces — a pairing that makes the jungle harder to detect and more expensive to untangle.

Key takeaways

• The orchestration jungle is the most common structural debt trap in production multi-agent systems.
• It emerges naturally from sequential agent chaining — the default pattern most teams reach for first.
• Every link in a sequential chain is both a latency penalty and a change-propagation risk.
• The fix is not better prompts but a different graph: parallel execution, deterministic gates, and scoped state.
• Distributed systems solved the same structural problem with the Saga pattern, DAG scheduling, and typed contracts. Those lessons transfer directly.

How orchestration jungles form

The pattern is predictable because it follows the path of least resistance.

Stage 1: The prototype. A single agent handles the task end-to-end. It works for the demo. Prompts are long, tool calls are many, and the agent sometimes gets confused — but it ships.

Stage 2: The split. The team breaks the monolithic agent into specialized sub-agents. One handles intake, another does retrieval, a third generates the output, a fourth runs validation. Each agent is simpler and more reliable on its own. The team calls this “good architecture.”

Stage 3: The chain. The sub-agents are wired sequentially. Agent A’s output is Agent B’s input. The chain is easy to reason about — data flows in one direction. But every agent adds latency, and every handoff is a point where format mismatches, missing fields, or unexpected values can break the downstream agent.

Stage 4: The jungle. Edge cases accumulate. Agent B sometimes calls back to Agent A for clarification. Agent D needs context from Agent A that Agent C did not pass through. Retry wrappers appear at each handoff. Shared state grows — first a simple dictionary, then a complex object that every agent reads and some agents write. Error handling becomes a patchwork that masks failures rather than resolves them.

At this point, the workflow has the structural properties of a jungle: dense coupling, hidden dependencies, and no clear path through.

This progression is not hypothetical. Hong et al. (2024) observed the same pattern in multi-agent software development — MetaGPT’s core contribution was replacing free-form agent chat with Standardized Operating Procedures (SOPs) that enforced structured handoffs. Without that structure, their multi-agent teams produced cascading errors at every handoff. The lesson: unstructured agent chaining does not scale, regardless of how capable the individual agents are.

Why sequential chains fail at scale

Sequential chains are the default because they mirror how humans think about multi-step processes. Every agent framework — LangChain, LangGraph, OpenAI Agents SDK — makes sequential chaining the easiest pattern to implement. Wu et al. (2023), in the AutoGen paper from Microsoft Research, explicitly designed for flexible multi-agent conversation patterns, but teams default to chains because they are conceptually simple.

But sequential chains have three structural weaknesses that get worse with scale.

Latency compounds

Each agent must wait for the previous one to finish. If each takes 3 seconds, a four-agent chain takes 12 seconds minimum — plus serialization overhead and retry logic at each handoff. Variance compounds too: the 95th-percentile latency of the chain is much worse than the 95th percentile of any single agent. Users experience this as unpredictable response times.

This is a well-studied problem in distributed computing. Topcuoglu et al. (2002) demonstrated with the HEFT algorithm that DAG-based task scheduling on heterogeneous processors consistently outperforms sequential scheduling — the same principle applies when your “processors” are LLM agents with varying latencies and capabilities. The compound AI systems community has reached the same conclusion: Zaharia et al. (2024) argue that production AI is shifting from standalone models to compound systems where orchestration is the central engineering challenge.

Change propagation is unbounded

Modifying Agent B’s output format requires updating Agent C’s input parsing, which may change Agent C’s output, which breaks Agent D. In a typed API, the compiler catches this. In a natural-language handoff between agents, the mismatch may not surface until production. This is prompt drift at the orchestration level — local changes with non-local effects.

Failure modes multiply

In a sequential chain of four agents, each with a 95% success rate, the chain’s success rate is 0.95⁴ ≈ 81%. Add retry logic and the effective rate improves, but at the cost of increased latency and stochastic tax. The more agents in the chain, the more you pay in retries to maintain acceptable throughput.

Chen et al. (2023) — the FrugalGPT paper — quantified a related problem: compound LLM calls amplify cost superlinearly, with multi-step chains consuming up to 98% more resources than necessary when every step uses the same high-capability model. The fix they proposed — routing each call to the smallest model that can handle it — applies directly to agent chains.

The distributed systems precedent

Tech leaders who built microservice architectures in the 2010s will recognize this pattern. The orchestration jungle is a distributed systems problem wearing an AI costume.

In 1987, Garcia-Molina and Salem introduced the Saga pattern at ACM SIGMOD: a long-running transaction broken into a sequence of local transactions, each with a compensating action that undoes its effect if a later step fails. The Saga was designed precisely for the problem agent chains face — a multi-step process where any step can fail and the system needs a principled way to recover.

The parallels are direct:

Distributed systems concept	Agent orchestration equivalent
Saga (compensating transactions)	Agent rollback on failure — undo tool calls, restore state
Choreography vs orchestration	Agent-to-agent messaging vs centralized controller
Circuit breaker pattern	Retry caps and graduated autonomy downgrades
Service mesh / API gateway	Model gateway with routing and fallback
Typed service contracts (gRPC, protobuf)	Tool schema contracts with JSON Schema validation
Idempotent operations	Deterministic gates that produce the same result regardless of retry count

The microservice world learned these lessons through a decade of production incidents. Agent systems can skip the repeat if teams treat multi-agent workflows as distributed systems from the start — because that is what they are.

The business impact

Orchestration jungles are expensive in three ways that compound:

Operating cost. Every retry, escalation, and timeout-driven failure is stochastic tax that a cleaner graph would not incur. The insurance workflow in the framework paper demonstrated this: converting a sequential chain of policy-specific agents into a parallel DAG reduced both latency and retry cost structurally — not through prompt tuning, but through graph redesign.

Velocity cost. When every change to one agent requires revalidating the entire workflow, deployment frequency drops. Teams route around the chain rather than modify it, which creates shadow implementations and duplicated logic.

Incident cost. When a jungle fails, diagnosing which handoff introduced the bug is expensive. The coupling means symptoms surface far from causes, and the shared state makes it difficult to reproduce the failure in isolation.

Diagnosing the jungle in your system

You may already have an orchestration jungle. These are the symptoms:

• End-to-end latency is 5–10x the sum of individual agent latencies. The extra time is handoff overhead, retries, and waiting.
• A change to one agent requires testing the entire workflow. You cannot deploy Agent B in isolation because its behavior depends on Agent A’s output format, which is undocumented.
• Shared state has grown beyond what any one team member can describe. If you cannot draw the state object on a whiteboard, the jungle has taken hold.
• Retry rates are climbing without code changes. This often means the handoff contracts are drifting — agents producing outputs that are technically valid but semantically different from what the downstream agent expects.
• The team is afraid to touch the orchestration layer. When developers route around the chain rather than modifying it, coupling has become prohibitive.

Five structural redesigns

The orchestration jungle is not fixed by tuning prompts or adjusting temperature. It is fixed by changing the graph.

1. Parallelize independent agents

If Agent B and Agent C do not depend on each other’s output, run them concurrently. This is the single highest-impact change: it cuts latency and reduces the failure surface by removing a sequential dependency.

Many graph engineering frameworks now support this natively. LangGraph’s fan-out/fan-in pattern, Google ADK’s parallel execution, and Microsoft Agent Framework’s concurrent steps all make parallelization a configuration change rather than an architectural rewrite.

2. Replace agent handoffs with typed contracts

Natural-language handoffs between agents are the equivalent of passing untyped dictionaries between functions. They work until they don’t, and when they break, the error message is “the agent did something unexpected.”

Define explicit schemas for what each agent produces and what the next agent expects. Validate at every handoff. This is the tool schema contracts governance control applied to inter-agent communication. Protocols like MCP and A2A are moving the industry toward standardized agent interfaces — a recent survey of AI agent protocols (Ehtesham et al., 2025) maps the landscape of these emerging standards.

3. Route each step to the right model

Not every step in a multi-agent workflow requires the most capable model. The FrugalGPT approach — routing each call to the smallest model that can handle it — applies directly. Classification, extraction, and formatting steps can run on smaller, faster models at a fraction of the cost and latency. A model gateway makes this routing transparent: the agent sends a request, the gateway picks the model.

4. Replace LLM calls with deterministic gates

Not every step in a multi-agent workflow requires an LLM. Routing logic, format validation, threshold checks, and policy rules can be expressed as deterministic code that runs in milliseconds with zero variance. Every step you move from the probabilistic path to the deterministic path reduces latency, eliminates a source of stochastic tax, and makes the workflow easier to test.

5. Scope shared state

A jungle’s shared state tends to become a god object — everything reads from it, some things write to it, and nobody knows which fields are stale. The fix:

• Each agent gets its own working memory, not a reference to a global context.
• Shared state is limited to explicit, versioned inputs and outputs at well-defined handoff points.
• Memory that persists across sessions is scoped per-agent or per-workflow, not global.

This mirrors the bounded-context principle from domain-driven design: each agent owns its data and exposes only what others need through a defined interface.

Prevention: principles that keep jungles from forming

It is easier to prevent an orchestration jungle than to untangle one.

Start with the DAG, not the chain. Before writing any agent, draw the dependency graph. Which agents depend on which outputs? If the graph is a straight line, ask whether it has to be — often two or three steps can run in parallel.

Define contracts before agents. Decide what each agent produces (schema, format, fields) before you write the prompt. This prevents the gradual contract drift that turns chains into jungles. Qian et al. (2024) demonstrated this in ChatDev — defining the communication protocol between agents before implementing them reduced cascading errors by enforcing structured phase transitions.

Budget for stochastic tax. Every agent you add to the graph increases your operating cost. Model the cost before you add the agent. If a deterministic function can do the job, use a function.

Set a depth limit. If your workflow requires more than three to four sequential agent steps, treat that as a design smell. Either parallelize, or ask whether one of those steps can be a deterministic gate instead.

Treat it as a distributed system. Apply the patterns that microservice teams learned over the past decade: circuit breakers, idempotent operations, compensating transactions, typed contracts, and centralized observability. These are not optional extras — they are the engineering minimum for any system where multiple autonomous components coordinate to produce a result.

From jungle to graph

The orchestration jungle is not a sign of bad engineering — it is a sign of successful prototyping that outgrew its architecture. Every production multi-agent system that started with “let’s just chain them together” will reach this point. Distributed systems hit this wall with microservices. Data engineering hit it with pipeline jungles. The structural problem is the same; only the execution model is new.

The question is whether you recognize it before the latency, cost, and fragility force a rewrite — or after. The agentic technical debt framework gives you the vocabulary to name it and the governance controls — golden-set evaluation, tool schema contracts, model gateways, graduated autonomy, and workflow graph redesign — to fix it.