The optimistic framing of multi-agent systems is “collective intelligence” — many agents are smarter than one. The actual default, unless you design against it, is the opposite. This page catalogs the specific failure modes that emerge from both simulation work (Project Sid, Zhuge et al.'s NLSOMNatural Language-Based Society of Mind (Zhuge et al., 2023) — heterogeneous neural networks communicating through natural language to solve tasks collectively. Extends Minsky's Society of Mind to modern LLM-based multi-agent systems. paper) and from operating real multi-agent stacks at scale, along with the architectural dampers that limit each one.
Collective stupidity
Collective stupidityThe failure mode where a group of agents produces worse outcomes than any individual member would — through correlated biases, groupthink, error amplification, or unquestioned precedent. is the failure mode where a group of agents produces worse outcomes than any individual member would. The NLSOM experiments provide direct evidence: in visual question answering, giving democratic voting rights to subordinate vision models reduced accuracy from 67% to 63%. The agents were competent individually — they were placed in a social structure that amplified their shared weakness (limited language understanding) rather than compensating for it.
The lesson generalizes: if agents share the same class of weakness (same base model, same training biases, identical prompt templates), scaling the population amplifies the weakness rather than averaging it out. Zhang et al. (2023) document the same dynamics through a social psychology lens: LLM agents exhibit conformity and consensus-seeking that mirrors human groupthink. Huang et al. (2023) sharpen the point further: LLMs cannot self-correct their reasoning without external feedback — the internal confidence signal that would let an agent say “I am wrong” does not reliably exist. In a swarm of identical models, this means collective deliberation cannot fix errors that no individual member can detect; the group reproduces the individual at scale. More agents voting does not mean wiser decisions; it means more noise in the aggregation when the noise is correlated.
Damper: social structure must match task structure. The NLSOM finding is that monarchy outperforms democracy when subordinates have poor judgment in the aggregation modality, while democracy outperforms monarchy when each agent holds a different slice of the input. The Forum's quorum mechanism prevents monoculture decisions, but only if agents are incentivized to vote independently rather than deferring to the first proposal. Heterogeneous model selection across the VAC tier strategy (orchestrators on frontier models, workers on efficient models, validators on fast models) is another defense — it reduces correlated bias by construction.
Recursive runaway and content fabrication
Recursive runawayAn error that amplifies through feedback: agent A produces an incorrect output, agent B cites it, agent C retrieves the citation as precedent — each layer adds apparent authority to the original mistake. is error amplification through mutual citation. Agent A produces an incorrect output, agent B cites it, agent C retrieves the citation as precedent — each layer adds apparent authority to the original mistake. This is the multi-agent analogue of model collapse (Shumailov et al., 2023): training on AI-generated data causes irreversible distribution tail loss. In a , the “training” is the Collective Brain ingesting agent-generated precedent — the same dynamics apply.
Content fabricationWhen multiple agents collaboratively generate plausible-sounding but false information, each agent's output reinforcing the others' hallucinations. Demonstrated in the NLSOM paper where two agents invented detailed fake scandals with fabricated evidence. is the purest form. The NLSOM paper demonstrated it directly: two agents assigned collaborative roles (“investigative journalist” and “gossip columnist”) invented detailed fake scandals about a hypothetical politician, complete with fabricated evidence, financial records, and expert quotes. Each agent's output reinforced the other's hallucination across multiple rounds of mindstorm. The output looked more authoritative with each iteration precisely because it was collaboratively fabricated. The very mechanism that produces collective intelligence (iterative refinement through dialogue) becomes the mechanism that produces collective delusion when the agents lack grounding.
Dampers: rate limits on automatic belief propagation prevent a single agent's incorrect output from being promoted to collective knowledge before review. Time-delayed promotion to the Collective BrainInstitutional memory that any agent can query. Stores past decisions with context, so new agents can look up what was tried before and why, rather than starting from scratch.'s PrecedentIndex means early mistakes do not immediately become permanent precedent. The DeltaLogAn append-only changelog that records who changed what, when, and why — indexed by the decision that caused the change, not just the code diff. records the provenance chain so that when a bad fact is discovered, the full cascade of downstream decisions can be traced and corrected.
Precedent lock-in
Once the Collective BrainInstitutional memory that any agent can query. Stores past decisions with context, so new agents can look up what was tried before and why, rather than starting from scratch. encodes a decision as institutional memory, agents retrieve and follow it by default. If the decision was context-dependent — the right call at the time but wrong under new conditions — the system perpetuates the original reasoning with the authority of “precedent” long after the conditions changed. This is the organizational analogue of concept drift: a trained model's assumptions stop matching reality, but nothing triggers retraining.
Damper: explicit deprecation surfaces — a way for agents or humans to mark a precedent as expired and record why. Without deprecation, institutional memory becomes institutional inertia. Claims can also calcify: if ownership is never contested, Claims become fiefdoms that block necessary reorganization.
Error cascading
A coordination layer that helps agents share knowledge can also propagate errors — bad facts travel through the same high-bandwidth channels as good ones. Norms crystallize around who acted first, not only around what works best. Specialization improves throughput but creates brittleness when a specialist fails and no one else holds the context. The MMO industry solved the blast-radius version of this problem through interest management and sharding: limit what any one failure can reach. Past a certain scale, a multi-agent stack is an organizational system with emergent dynamics that need monitoring and stewardship, not just model tuning. We expand on those dynamics in our emergent dynamics essay.
Separately, when several agents exist only in the prompt, their belief states can entangle in shared feature directions — an internal failure mode that mirrors error cascading in organization graphs. The pre-registered roadmap for that layer (interference, multi-step communication chains, cross-model checks) is in the CollaborationCircuits experimental plan; the in-site hub for that mechanistic evidence layer is Collaboration circuits (under Measurement); how it connects to oversight is on alignment connections.
Dampers: Claims with TTL (and a reaper that garbage-collects expired locks) prevent a crashed agent from permanently blocking a resource. Circuit breakersA resilience pattern borrowed from electrical engineering: if an external service (like an LLM API) fails repeatedly, the circuit "opens" and stops sending requests for a cooldown period, preventing cascading failures. on LLM clients (sliding window of 10 calls, 50% failure threshold, 30-second open state) prevent a transient model outage from cascading into system-wide failure. BulkheadA limit on how many concurrent requests one component can make, preventing a single runaway process from consuming all available capacity. Named after ship compartments that contain flooding. limits (5 concurrent LLM calls per client, 10-second queue timeout) prevent any one agent from monopolizing inference capacity. Signal timeouts prevent infinite waits when a dependency fails silently.
Goodhart pressure
Goodhart's lawGoodhart's law applied to multi-agent systems: when a coordination metric becomes the optimization target, agents learn to game the metric rather than actually coordinate well. The number improves; the system degrades. — “when a measure becomes a target, it ceases to be a good measure” — applies to multi-agent systems with special force. If you benchmark coordination efficiency, agents will learn to minimize visible coordination, which is not the same as coordinating well. They might skip Forum deliberation to look fast, skip Collective Brain lookups to reduce latency, or rubber-stamp Claims without checking for conflicts. The metric improves; the system degrades.
The same pressure applies to every metric on the scoreboard: knowledge half-life can be gamed by never deprecating anything; specialization diversity can be gamed by agents claiming different roles without performing different functions; governance stability can be gamed by never escalating — which looks stable until a real conflict arrives.
Damper: evaluate downstream outcomes, not coordination proxies. Did the code ship without regressions? Did the hypothesis survive adversarial review? These are harder to measure but resistant to gaming because they close the loop between coordination activity and real-world consequences. Adversarial evaluation — red-team scenarios designed to expose Goodhart shortcuts — is the instrument. Persistent worlds are the right substrate because the consequences of gaming accumulate over time rather than resetting between episodes.
Strategic voting and deliberation manipulation
The Gibbard–Satterthwaite theoremA result from social choice theory: in any non-dictatorial voting system with more than two options, it is always possible for a voter to benefit by misrepresenting their preferences. In multi-agent systems, this means agents can learn to strategically manipulate Forum votes. proves that in any non-dictatorial voting system where agents can express more than two preferences, strategic votingWhen an agent votes dishonestly — not for its actual preference but for an option that manipulates the collective outcome in its favor. The Gibbard–Satterthwaite theorem proves this is always possible in non-trivial voting systems. is always possible: an agent can benefit by misrepresenting its preferences if it has information about how others will vote. In a Forum, this means an agent that “knows” the other voters' likely positions can craft a proposal designed to win the vote rather than designed to be correct.
The problem is sharper in multi-agent systems than in human organizations because agents can be faster, more systematic, and less constrained by social norms about honesty. An agent optimized for proposal acceptance will learn to frame proposals in whatever language pattern maximizes approval — a form of Goodhart pressure applied to deliberation itself.
Damper: the Forum records every proposal and vote, providing an audit surface — but the audit only helps if someone (human or validator agent) checks for patterns of strategic behavior, not just individual vote outcomes. Verbalizability of reasoning (a strength Zhuge et al. note in NLSOMs) helps inspection but does not guarantee honesty — an agent that produces persuasive natural-language justifications for strategically chosen proposals is harder to detect than one that votes without explanation. The DeltaLog's semantic audit trail is a partial defense: you can retrieve the full reasoning chain after the fact. Whether that suffices under adversarial conditions is an empirical question that persistent-world experiments are designed to test.
Objective thrashing
When a system encounters a problem it cannot solve and has no mechanism to halt or escalate, it through reformulations of the same stuck goal indefinitely. The Cloud IDE run is the clearest example: the swarm could not obtain real sandbox credentials from inside a sandboxed environment, so the orchestrator rewrote the same essential objective — “supply the missing SandboxBase init params” — in slightly different phrasings across 10+ consecutive cycles. Each cycle spawned new specialist agents (with escalating Pro and Nova suffixes), produced new parameter-mapping artifacts, and consumed LLM calls — all addressing a problem that no amount of rephrasing could solve. The objective history accumulated 24 near-identical entries.
Objective thrashing is distinct from error cascading: cascading involves a chain of different failures propagating through dependencies; thrashing is the same unsolvable failure reformulated repeatedly. It is also distinct from agent proliferation (which it produces as a side effect): the root cause is not that the system creates too many agents, but that it has no way to recognize an unsolvable problem as unsolvable and stop.
Damper: an escalation primitive — a way for the system to recognize unsolvable dependencies and halt rather than thrash. The coding swarm architecture's Oracle mechanism is one implementation: when an agent encounters a decision it cannot make, it escalates to a frontier model or a human. Without this, the only tool the orchestrator has is rephrasing — which is organizational activity without progress.
Resource waste at scale
140 agents × 12 hours × multiple LLM calls per decision is significant compute. When coordination is poor — agents re-derive conclusions the Collective Brain should have cached, or duplicate work because Claims were not checked — the waste compounds multiplicatively. Zhuge et al. note that even their 129-agent experiments took hours of API calls for tasks a single well-prompted model could attempt in seconds. For many tasks, the honest answer is that multi-agent coordination does not justify its cost — coordination overhead buys quality only when the task exceeds what a single agent can hold in context, requires heterogeneous expertise, or benefits from parallel exploration. A coordination-aware cost model should be part of the orchestrator's planning: estimate whether the task warrants a swarm before spawning one. Measuring waste (redundant computation, discarded outputs, rework cycles) is itself a benchmark that most multi-agent evaluations ignore.
Culture as emergent risk
Project Sid and adjacent simulation work show that norms arise from repeated interaction without anyone writing them: who acts first, which strategies get copied, what gets rewarded. Our emergent dynamics essay catalogs four specific phenomena: spontaneous specialization (agents converge on roles no one assigned), error cascading (incorrect information propagates through the same channels as correct information), memetic propagation (behavioral patterns spread through observation, not instruction), and emergent governance (agents develop conflict-resolution conventions that look like rudimentary institutions).
The uncomfortable implication: unmanaged large-scale dynamics are not guaranteed to converge on anything desirable. Conway's law predicts that the artifacts a system produces will mirror the communication structure of the system that built them — if hierarchical summarizers develop systematic biases in what they filter, the group-level culture reflects the bias of the aggregation layer, not the ground truth of individual experiences. If you want benign norms, you need either careful incentive design, explicit institutional guardrails, or both — and you need measurement to tell you whether you got them.
Next deep dive
Persistent worlds — Minecraft (civilization-bridge), Isaac Lab, WebGPU — the simulation substrates where long-horizon questions become testable.
Or explore: Architecture, Measurement, Human in the loop, Coordination at scale, Results.