Multi-agent AI systems are fundamentally unpredictable and that’s exactly how we should build them.
The AI industry has a dirty secret: we’re building systems we can’t fully understand, predict, or control. Multi-agent LLM architectures, agent swarms exhibit behaviors that emerge from interactions rather than individual programming. The instinct is to fight this, to seek provable guarantees and predictable outcomes.
But there’s a better path: embrace uncomputability as a feature.
The Busy Beaver Meets AI
The Busy Beaver function BB(n) asks: what’s the maximum computation an n-state Turing machine can perform before halting? It’s uncomputable - no algorithm can calculate it for arbitrary n. This isn’t implementation complexity; it’s fundamental impossibility.
Multi-agent AI systems face the same barrier. When N agents interact, each with internal states and the ability to communicate, you’re asking: what’s the maximum useful work they can produce before reaching consensus or diverging? Call it BB(N, n_states). Like its ancestor, it’s uncomputable.
Erlang - Ghost in the Machine
The late Joe Armstrong, creator of Erlang, taught us something profound: “You cannot know the state of a distributed system; you can only know what happened.”
In Erlang’s process model, global state is unobservable. Each process has local state, messages flow between them, but the complete system state at any instant is fundamentally unknowable.
This maps perfectly to multi-agent AI:
Each LLM agent has local context (computable).
They exchange messages, prompts, tool calls (observable).
But their collective “understanding” or emergent strategy? Uncomputable.
Erlang’s philosophy accepts that processes will fail unpredictably. Instead of preventing failures, you architect for recovery: supervision trees, isolation boundaries, explicit message passing, and observability over omniscience.
So, just stop trying to prove your multi-agent system is correct. Instead:
Observe Everything. You can’t know the state, but you can log what happened. Implement comprehensive tracing: message passing, agent decisions, token consumption, temporal causality graphs. Build the equivalent of Erlang’s process monitors for AI agents.
Bound Explicitly. Since you can’t compute resource needs, set hard limits: maximum reasoning steps, token budgets, wall-clock timeouts, recursion depth. Accept that you’re trading completeness for tractability.
Isolate Failures. Each agent should be a supervised entity with clear failure boundaries. When one agent halts or produces garbage, it shouldn’t cascade. This is Armstrong’s supervision trees applied to AI: hierarchical oversight with restart strategies.
Design for Emergence. You can’t predict what strategies N agents will discover, so don’t try. Instead, create fitness landscapes and selection pressure. Use evolutionary approaches to explore the uncomputable space of agent configurations. Test, observe, iterate - but never assume you’ve found the optimum.
Keep Humans in the Loop. For critical decisions, accept that uncomputability means you need human judgment. The goal isn’t full automation; it’s augmentation with appropriate guardrails. And guess which unit of the system needs more guardrails? ;)
So let’s rephrase Joe’s saying for AI engineering: “You cannot predict multi-agent AI; you can only architect for its emergence.”
The Architecture of Acceptance and Upside of Impossibility
We accept that programs have bugs, distributed systems have network partitions, and hardware has failure rates. Now we need to accept that multi-agent AI has uncomputability.
Embracing uncomputability liberates you. You stop seeking the perfect agent configuration and start exploring the space of “good enough” solutions. You stop trying to prove safety and start building testable safety boundaries. You stop chasing theoretical guarantees and start shipping systems with empirical track records. Most importantly, you align with reality. BB(N, n) grows faster than any computable function. Your multi-agent system can produce results you couldn’t ever dream of before.
The Busy Beaver doesn’t apologize for being uncomputable. Neither should your AI agent architecture.