Most multi-agent papers report aggregate metrics: task success rate, cost, latency. This page reports something harder to measure and more revealing — the organizational dynamics of a real run. The objective was open-ended: “build a photonic deep-learning compiler startup and do R&D on it.” The system received no domain expertise. What it produced — and how it broke — illustrates the coordination primitives and failure modes described elsewhere on these pages, now grounded in data rather than theory.
The run
A single session, starting from a one-sentence objective. The model tier strategyAssigning different model sizes to different roles based on decision impact: orchestrators (~5% of decisions) use frontier models for high-stakes strategy, workers (~50%) use efficient models for bulk execution, validators (~45%) use fast models for high-frequency checks. Reduces cost while concentrating capability where it matters most. used GPT-5-nano as the powerful tier and Exaone-3.5-32B-Instruct (via Together) as the fast/cheap tier. The system began with 14 core agents — Orchestrator, ChiefOfStaff, ChiefAgentOfficerA meta-agent responsible for hiring: it analyzes the objective, determines what specialist roles are needed, and generates design specs for new agents that the AgentDesigner then implements., AgentDesignerA meta-agent that takes a role specification from the Chief Agent Officer and generates the actual agent code — Python classes with domain-specific prompts, tool access, and behavioral contracts., CodeVerifier, ContextCompressor, and others — and dynamically generated 139 specialist agents over the course of the run. 38 agents were active (received tasks and made LLM calls). The session produced 514 LLM calls over ~2.5 hours of API time, generated 57 artifacts, logged 196,829 structured events, and completed 9 stages of phase-based executionA structured agent work cycle: understand → plan → execute → verify → complete. Gate-chain validation between phases catches errors before they propagate to the next stage..
The stages tell the story of the system's organizational evolution:
- Foundation & Telemetry Baseline — Produce observability infrastructure and initial architecture. Two pods: an Observability Setup pod (telemetry schema, baseline metrics) and a Compiler Architecture Foundation pod (7 agents: Optical Compiler Architect as pod lead, plus hardware abstraction, kernel mapping, optimization, dataflow, validation, and photonic simulation specialists). Delivered 8 foundational artifacts including
TelemetryBaseline.json,MVP_Architecture_Spec,IR_Design_Spec,Kernel_Mapping_Strategy, andDataflow_Interface_Def. - MVP End-to-End Prototype — Wire IR → kernel mapping → dataflow → hardware backend for a toy DL model. This stage triggered the first re-plan (see below).
- Stage 0: Validation & Gating for Stage 3 — Formal validation of the end-to-end MVP. Outcome: “partial_success_with_blockers” — the system found its own gaps and could not proceed.
- Stages 3–4 — Orchestrator-initiated recovery: telemetry gating metadata alignment, hardware readiness checks, and Stage 2 output refresh.
- Hardware-Backed MVP Readiness — Final milestone plan and readiness validation. Produced
Hardware_Readiness_Decision.jsonwith a “ready for MVP hardware deployment” status.
Spontaneous specialization and agent self-design
The system began with no domain knowledge about photonic computing. The Chief Agent OfficerA meta-agent responsible for hiring: it analyzes the objective, determines what specialist roles are needed, and generates design specs for new agents that the AgentDesigner then implements. analyzed the objective and produced a hiring plan for 16 initial domain specialists: Optical Compiler Architect, Photonic Hardware Abstraction Engineer, DL Kernel Mapper Specialist, Domain-specific Optimization Scientist, Dataflow & Throughput Engineer, Security & Threat Modeling Engineer, and others. The Agent DesignerA meta-agent that takes a role specification from the Chief Agent Officer and generates the actual agent code — Python classes with domain-specific prompts, tool access, and behavioral contracts. then generated each specialist as a Python class with domain-specific prompts, tool access patterns, and behavioral contracts — 178 LLM calls, zero failures, 11.4 seconds average.
As the run progressed, the orchestrator discovered needs the initial hiring plan did not anticipate. New agents were spawned on demand: Telemetry Baseline Designer, Stage3 Revalidation Architect, Hardware Readiness Signals Engineer, MVP Prototype Readiness Auditor, Artifact Readiness Analyst, Data Provenance Auditor. The final roster of 139 agents was not designed upfront — it emerged from the system encountering problems that existing agents could not solve. This is spontaneous specialization in the precise sense: roles that no one assigned, created by the organizational system's encounter with task structure.
The specialization was not uniform. The AgentDesigner (178 calls) and Orchestrator (165 calls) dominated the call volume — meta-coordination consumed 67% of all LLM calls. The domain specialists who actually built artifacts made far fewer calls: IR Design Architect (4), DL Kernel Mapper Specialist (3), Optical Compiler Architect (2). The organizational overhead of figuring out what to do dwarfed the cost of doing it. This is a real-world instantiation of the coordination overhead question the measurement page describes: the plumbing works, but the ratio of coordination to production matters.
Artifact creation dynamics
The run produced 57 artifacts with full lineage trackingA directed graph recording which artifacts depend on which other artifacts, who produced them, and which stage they belong to. Used for traceability and to understand the provenance chain when something goes wrong. — a directed graph recording which agent created each artifact, which agents consumed it, and which stage it belonged to. The lineage graph reveals the organizational structure that emerged:
Hub artifacts dominated the dependency graph. MVP_Architecture_Spec was read by 27 downstream agents — it became the de facto coordination document, not because anyone designated it as such, but because every specialist needed architectural context to do their work. Similarly, IR_Design_Spec, Kernel_Mapping_Strategy, Optimization_Goals, and Dataflow_Interface_Def each fanned out to 25+ consumers. These five artifacts, all produced in Stage 1, defined the vocabulary and contracts that every later stage built on.
This is Conway's law made visible: the artifact dependency graph is the communication structure. When MVP_Architecture_Spec contained an ambiguity, every downstream agent inherited it. When Dataflow_Interface_Def was revised in Stage 2 (producing Dataflow_Interface_Spec_Revision.json), the revision had to propagate through the same 25+ consumers — and any agent that had already committed to the original interface spec now held stale context. The precedent lock-inWhen a past decision encoded in institutional memory continues to guide behavior after the conditions that justified it have changed. The organizational analogue of concept drift in ML models. dynamic is not hypothetical in this data: early artifacts became load-bearing institutional memory, and correcting them required explicit re-validation stages.
Metadata artifacts proliferated. Of the 57 artifacts, 17 were .meta.json files — metadata about metadata. The system generated provenance records, gating assessments, traceability matrices, ontology maps, and validation reports about other validation reports. This is a form of organizational overheadThe failure mode where a group of agents produces worse outcomes than any individual member would — through correlated biases, groupthink, error amplification, or unquestioned precedent. that emerged without anyone requesting it: the validator agents, acting on their role prompts to “ensure correctness and schema conformance,” produced audit artifacts that consumed agent capacity without advancing the core objective. Nearly 30% of the artifact output was self-referential governance — a concrete example of coordination primitives generating more coordination rather than more value.
The write collision: re-planning under failure
The most instructive moment in the run was a write collision during Stage 2. Two pods — “IR-Kernel-Dataflow-Prototype” and “Validation, Telemetry & Reporting” — both attempted to write ToyModelValidationResults.json. The system detected the conflict and triggered a re-planning event. The orchestrator received the feedback:
“Artifact write-collision detected for ‘ToyModelValidationResults.json’. Ensure unique outputs per pod or coordinate write order.”
The orchestrator responded by merging the two pods' validation responsibilities into a single pod with explicit artifact ownership — the IR-Kernel-Dataflow-Prototype pod expanded to 9 agents with the IR Design Architect as pod lead, and the validation outputs were de-duplicated. This is exactly the dynamic the Claims primitive is designed to prevent: without exclusive ownership of artifact names, parallel pods produce conflicting writes. In this run, the system recovered through re-planning rather than through Claims (this was an earlier version of the stack), but the failure mode is the one that motivated the Claims design.
The re-plan cost was non-trivial: the orchestrator generated a new stage planThe orchestrator's output for a single execution stage: a set of pods with assigned agents, expected artifacts, dependencies, success criteria, and identified risks. Plans are generated JIT, one stage at a time. with revised pod structure, reassigned agents, and restarted execution — elapsed time that would have been avoided with upfront artifact ownership. But the system did recover autonomously. The orchestrator did not crash, did not produce corrupted output, and did not require human intervention. It diagnosed the problem, restructured the organization, and continued. That recovery path — detect conflict, re-plan, resume — is the JIT planningJust-in-time planning — the orchestrator plans one stage at a time using current information, rather than committing to a rigid upfront plan. Adapts to new information as it arrives. loop working as designed.
The gating cascade: how blockers propagate
Stage 0 (“Validation & Gating for Stage 3”) ended with status “partial_success_with_blockers.” Five blockers were identified:
TelemetryBaseline.json: gating metadata missingTelemetry_Instrumentation_Plan.json: gating metadata missingHardware_Backend_Interface_Def: hardware readiness pending milestoneToyModel_Buffer_Config.json: buffer config in-progress, latency modeling pendingPhase0_E2E_Validation_Report.json: e2e validation in-progress due to telemetry gating dependencies
Notice the dependency chain: the telemetry baseline was missing gating metadata, which blocked the instrumentation plan, which blocked the e2e validation report, which blocked the gating decision. A single upstream gap — metadata fields not included in the original telemetry baseline — into a system-wide hold. The system responded by generating three additional stages (Stages 3, 4, and the Hardware Readiness Kickoff) specifically to close these gaps.
This cascade is the error cascading failure mode operating through artifact dependencies rather than through information propagation. The original telemetry artifact was not wrong — it was incomplete, and the incompleteness was invisible until a downstream consumer tried to use the missing fields. The Phase 0 E2E Validation Report explicitly flagged the readiness status as “At-Risk” and generated a remediation plan with five acceptance criteria, three gap descriptions, and concrete milestones with due dates. The validators did their job — the question is whether the 3-stage recovery was the cheapest path, or whether upfront schema contracts on the telemetry baseline would have caught the gap at Stage 1.
Meta-agent failures
Two infrastructure agents failed systematically. The Chief Agent OfficerA meta-agent responsible for hiring: it analyzes the objective, determines what specialist roles are needed, and generates design specs for new agents that the AgentDesigner then implements. — the agent responsible for analyzing the objective and producing hiring plans — failed on all 9 of its calls, with an average response time of 47 seconds per attempt. It eventually succeeded through a pagination fallback (1 call, 21 seconds, successful). The bottleneck was output size: the hiring plan for 16 specialists with detailed role descriptions exceeded the model's reliable output window, and the structured JSON parsing failed repeatedly before the system decomposed the task into smaller chunks.
The ContextCompressor — responsible for compressing long context histories into summaries — on all 8 of its calls. Average prompt was 6,445 tokens with a 398-token completion target. The failures suggest the compression task was underspecified: the agent received too much context relative to its output budget and could not produce valid compressed output. This is a resource-waste failure — 8 calls × 16.6 seconds average = 133 seconds of API time that produced nothing.
In contrast, the Optical Compiler Architect succeeded on 1 of 2 calls but with a 529-second average response time — one call took nearly 9 minutes. The Domain-specific Optimization Scientist took 509 seconds for a single call. These are the frontier-model agents handling the hardest reasoning tasks, and their latency is the binding constraint on the system's throughput. The model tierAssigning different model sizes to different roles based on decision impact: orchestrators (~5% of decisions) use frontier models for high-stakes strategy, workers (~50%) use efficient models for bulk execution, validators (~45%) use fast models for high-frequency checks. Reduces cost while concentrating capability where it matters most. distinction is visible in the data: the Orchestrator averaged 6.5 seconds per call (fast tier, 165 calls), while specialist agents on the powerful tier averaged 30–500+ seconds per call.
The readiness decision: institutional memory in action
The run's final output was Hardware_Readiness_Decision.json — a structured decision document that synthesized inputs from four upstream artifacts (Stage3 gating readiness, telemetry validation, gating alignment, and readiness provenance). The decision status: “ready for MVP hardware deployment” with risk level “low.” It listed four criteria met (telemetry alignment validated, gating alignment matches hardware criteria, provenance demonstrates full traceability, no unresolved gating gaps) and zero criteria missed.
The Stage3_Gating_Readiness_Consolidated.json artifact is more revealing. It assessed 6 subsystems with gap-level detail: the Compute Backend Interface was “Partial” readiness with a high-risk gap (no hardware test harness for backend compute load) and three mitigations with owners. Each subsystem had milestones, due dates, and explicit status tracking. This is what institutional memory looks like when it works: not a flat knowledge base, but a structured decision record with provenance, traceability, and explicit gaps that downstream stages can act on.
But the readiness decision also illustrates Goodhart pressureGoodhart's law applied to multi-agent systems: when a coordination metric becomes the optimization target, agents learn to game the metric rather than actually coordinate well. The number improves; the system degrades. in miniature. The system was optimizing for a “ready” verdict — and it produced one by closing every gap it could identify within its own artifact graph. Whether the artifacts accurately reflect reality (there is no actual photonic hardware) is a question the system cannot answer from inside itself. The validators validated the schema, the provenance, and the inter-artifact consistency — but no agent verified that the IR_Design_Spec actually describes a buildable photonic compiler. The system can confirm its own internal coherence but not its external validity. That gap is where human judgment remains load-bearing.
By the numbers
| Metric | Value |
|---|---|
| Agents generated | 139 |
| Agents active (received tasks) | 38 |
| Total LLM calls | 514 |
| Successful / failed | 445 / 69 (13.4% failure rate) |
| Total API time | 8,992 seconds (~2.5 hours) |
| Average call duration | 17.5 seconds |
| Average time to first token | 14.2 seconds |
| Stages completed | 9 (including 3 recovery stages) |
| Re-planning events | 1 (write collision) |
| Artifacts produced | 57 (including 17 .meta.json) |
| Structured events logged | 196,829 |
| Codegen requests / success / errors | 155 / 129 / 26 |
| Sandbox executions (E2B) | Logged via e2b.jsonl, Python 3.11, no network |
| Meta-coordination call share | 67% (AgentDesigner + Orchestrator) |
| Orchestrator avg latency | 6.5 seconds (fast tier) |
| Specialist avg latency | 30–529 seconds (powerful tier) |
What the run teaches
Coordination overhead is the dominant cost, not inference. Two-thirds of all LLM calls went to meta-coordination (designing agents, planning stages, compressing context) rather than to domain work. The photonic compiler specialists who produced the actual architectural artifacts made a small fraction of the total calls. This is an organizational-design problem, not a model-quality one. The question is whether the coordination overhead can be reduced without losing the adaptability that let the system recover from the write collision and gating cascade.
Artifacts are institutional memory — including the bad kind. Hub artifacts like MVP_Architecture_Spec became load-bearing coordination documents that every downstream agent depended on. When they were incomplete (missing gating metadata), the incompleteness propagated silently until validation stages surfaced it. This is the precedent lock-in problem operating at the artifact level: early decisions crystallize into context that later agents trust without re-deriving.
The system can diagnose and recover, but recovery is expensive. The write collision led to re-planning; the gating blockers led to three recovery stages. Both are autonomous corrections — no human intervened. But each correction consumed additional stages, agent capacity, and LLM calls. Upfront contracts (artifact schemas, Claims on output names, required metadata fields) would have caught both problems cheaper. The coordination primitives exist precisely to make these corrections unnecessary by preventing the conditions that require them.
Internal coherence is not external validity. The system produced a “ready for hardware deployment” decision backed by provenance chains, gating assessments, and validation reports. Every artifact references other artifacts; the graph is internally consistent. But the artifacts describe a photonic compiler that does not exist on real hardware. The system can validate its own organizational outputs but cannot ground them in physical reality. That boundary — where institutional memory ends and empirical verification begins — is where human oversight is non-negotiable, and it is the boundary the DeltaLog and Forum are designed to make visible.