Language models solve calculus and write compilers, yet struggle with negotiation, deception, and spontaneous cooperation. Lacuna's research direction on benchmarking social intelligence in multi-agent systems collects the evidence: evaluation is shifting from isolated cognition to agents as social actors inside multi-agent ecosystems — and the results are unflattering.
Winning is not persuading
The sharpest example is Meta's Cicero. Analyzed in More Victories, Less Cooperation, Cicero posts elite win rates in Diplomacy but lacks human-like persuasion — its play is transactional and bot-like, easily identified by experienced players. Lacuna calls this the performance nuance paradox: the win-loss metric is tautological. It certifies that the agent won, not that it understood anything about the humans it beat.
The conjectures saw this coming
This is the Bhatt Conjectures applied to a new domain. A Diplomacy win rate is a benchmark score; social mastery is the capability claim. The conjectures' framework says the score is necessary but not sufficient — and Lacuna's direction shows exactly where sufficiency fails. T1 asks whether capability survives a change of surface form; Cicero's persuasion collapses the moment a human partner deviates from expected play. TU asks for causal fidelity and metacognition; a transactional style that experienced players flag as bot-like is precisely the absence of a truth-preserving model of other minds. The conjectures predict that performance-based social benchmarks will overclaim, and the multi-agent literature is now supplying the counterexamples.
Theory of Mind is the missing engine
Social intelligence runs on Theory of Mind — attributing beliefs, intents, and desires to others. ToMBench shows even top models trail humans on deep social inference, and INTERINTENT finds a 20% human–AI gap in intention understanding in Avalon: inferring hidden roles from conversational cues. Surface linguistic pattern-matching passes static tests and fails live games.
From static tests to sandboxes
The field is responding with action-level environments: TextArena scores deception and persuasion in games like Werewolf, StarDojo tests long-term social interaction in Stardew Valley, and STSS asks whether agents can complete social goals — organizing an event — in a shared simulation. ProMediate goes further: a social agent must sense group dynamics and intervene proactively, not just respond to prompts.
Scaling does not close the gap
Single-Agent Scaling Fails Multi-Agent Intelligence evaluated 41 models and found that getting better at math and coding does not produce native social capability. Social intelligence appears to be a distinct dimension — which is why Lacuna points to ability-oriented measurement, comparative psychology, and environments like Animal-AI that test the building blocks rather than the performance.
The assurance lesson
For Shrewd Research, the pattern is familiar: a win rate is an observation; social competence is a claim. The evidence envelope for a social agent should record which partners it persuaded, how its reasoning was probed, where its model of other minds broke — not a leaderboard position. Whether agents reason like us or merely mirror us is an empirical question with a claim boundary, and the benchmark's job is to expose it.