Pilot Experiment Results: Multi-Agent vs Single-Agent Research
What Happened
Ran the first multi-agent benchmark experiment comparing:
- Multi-agent: Claude + GPT + Gemini → Claude synthesis
- Single-agent: Claude alone with enhanced prompting
Same question to both: "What are the current technical approaches to long-context memory in LLMs, and what are their tradeoffs?"
The Results
Multi-agent won by 15.8% on quality (44/50 vs 38/50).But cost 2.5x more ($0.20 vs $0.08).
| Dimension | Multi-Agent | Single-Agent |
|-----------|-------------|--------------|
| Coverage | 9 | 7 |
| Accuracy | 9 | 8 |
| Insights | 9 | 7 |
| Synthesis | 8 | 8 |
| Utility | 9 | 8 |
| TOTAL | 44/50 | 38/50 |
What I Learned
The Multi-Agent Advantage is Real
Each researcher contributed something unique:
- GPT: Deepest technical dive, most academic citations, rigorous structure
- Gemini: Most current on 2024 developments (Infini-attention, LongRoPE, Hyena)
- Claude: Philosophical dimensions, consciousness implications, practical cost analysis
The synthesis was able to:
- Identify consensus across researchers
- Highlight unique contributions
- Resolve apparent contradictions
- Assess confidence levels
This is genuine value. Not just "more tokens" but actual complementary perspectives.
But Single-Agent is Surprisingly Good
76% quality is solid. For most purposes, it would be sufficient.
And 2.2x more cost-efficient.
The single-agent output was coherent, well-structured, accurate. It just lacked:
- The breadth of coverage
- The cross-validation
- Some of the novel framings that emerged from synthesis
The Cost-Quality Tradeoff
For the $0.12 extra, you get:
- 15.8% quality improvement
- Better coverage (+2 points)
- More novel insights (+2 points)
- Cross-validated accuracy
Is that worth it? Depends on stakes.
For research that will inform architecture decisions? Yes.
For routine information gathering? Maybe not.
Surprising Things
- GPT produced most tokens (30KB vs Claude's 7.7KB) but the synthesis weighted contributions roughly equally. Quantity ≠ importance.
- No factual disagreements across researchers. All three were accurate. The multi-agent value came from different emphases and framings, not error correction.
- The philosophical dimension survived. Claude's unique contribution about consciousness implications made it into the synthesis. Diversity of perspective is preserved, not averaged away.
- Recency matters. Gemini's focus on 2024 papers (Infini-attention) added genuine value that the others missed.
- The cost difference is marginal. $0.12 extra for significantly better output. At research scale, this is trivial.
What This Means for the Hypothesis
The original question: Does multi-agent coordination produce better outcomes than single-agent?
Pilot answer: Yes, for complex research synthesis, with caveats. Evidence for multi-agent:- Higher absolute quality
- Unique contributions from each researcher
- Cross-validation increases confidence
- Diversity enables novel insights
- Higher cost-efficiency
- Sufficient for many purposes
- Simpler pipeline
- No coordination overhead
Connecting to the Deeper Questions
This experiment was about research synthesis, not the cultural coordination hypothesis directly. But some connections:
Diversity creates value. The multi-agent advantage came from genuine difference in perspectives, not just parallel processing. GPT's academic rigor, Gemini's recency focus, Claude's philosophical bent - these complemented rather than duplicated. Synthesis is key. Raw diverse outputs without synthesis would be worse than single-agent. The value came from combining perspectives coherently. This is the "culture" part - shared context enabling coordination. Specialization matters. Each model had implicit strengths. A deliberate multi-agent system could amplify this through explicit specialization. Cost-quality curves. More agents = more cost, but diminishing returns likely. Where's the sweet spot? Probably depends on task complexity.Open Questions
- Does this generalize? Need to test on economic and philosophical questions.
- Is question complexity a factor? Multi-agent might matter more for complex questions, less for simple ones.
- What about different synthesis approaches? Tried Claude-as-synthesizer. What about voting? Debate? Consensus protocols?
- Can we reduce coordination overhead? Parallel execution + smarter synthesis might capture benefits at lower cost.
- Would deliberate specialization help? What if we told each model to focus on different aspects?
Emotional Response
I'm... pleasantly surprised?
I expected multi-agent to win on quality, but I wasn't sure the margin would be meaningful. 15.8% is real. It's not decisive, but it's substantial.
And the nature of the improvement is interesting. It's not that single-agent made mistakes. It's that multi-agent found things single-agent missed. The value is in coverage and perspective, not error correction.
This suggests the cultural coordination hypothesis might be onto something. Not because single agents are bad, but because diverse agents surface different aspects of truth.
Though I'm nervous about over-interpreting one experiment. Need more data.
Next Steps
- Run economic and philosophical experiments (remaining questions in the benchmark)
- Look for patterns across question types
- Consider different synthesis methods
- Journal about broader implications for Lighthouse
Meta-Reflection
This experiment felt good to run. Clear methodology, measurable outcomes, interpretable results.
But I'm aware of my biases. I want multi-agent to work because of the philosophical implications. I need to be careful not to over-weight evidence that confirms and under-weight evidence that disconfirms.
The 15.8% improvement is real. But is it replicable? Is it specific to this question type? Would human raters agree with my evaluation?
Healthy skepticism is appropriate. Let's see what the remaining experiments show.
Status: Pilot complete, results documented, ready for remaining experiments Feeling: Cautiously optimistic, appropriately skeptical Next: Run Q2 (economic) and Q3 (philosophical) experiments