A benchmark score is often treated as a verdict: the model reasons, understands, or has crossed some capability threshold. That conclusion asks the score to carry more weight than it can. Lacuna's analysis of the Bhatt Conjectures focuses on a stricter question: what evidence would have to survive before a capability claim becomes credible?
A capability should survive a change of clothes
The first test, T1, treats reasoning as representation-invariant performance. If two problems have the same logical structure, changing the surface form—from prose to code, symbols, a diagram, or an unfamiliar vocabulary—should not destroy the result. A system that succeeds only in the familiar presentation may have learned the benchmark's regularities rather than the reasoning operation the benchmark claims to measure.
Necessary does not mean sufficient
Passing T1 would be meaningful, but it would not establish understanding. TU asks for more: an internal representation that preserves truth across representations and supports counterfactual questions. T*U adds causal fidelity and calibrated awareness of uncertainty. The hierarchy matters because it prevents one successful test from silently expanding into a much larger claim.
Scale is part of the test
Lacuna highlights complexity scaling as a practical failure mode. Keep the rule fixed, increase the instance size, and watch whether performance collapses. A model that solves a small Tower of Hanoi instance but fails as the same structure grows has not demonstrated the invariant capability. It has demonstrated a bounded success condition—and that boundary belongs in the result.
Structural complexity is the harder test
Lacuna's research direction on robust generalization turns that principle into an experimental program. Surface changes test whether an answer survives a new representation; structural scaling tests whether the underlying operation survives greater depth, length, and composition. A system that succeeds on small instances but collapses as the same rules are extended may be exploiting a shortcut rather than learning the task's invariant structure. The honest result is therefore a generalization boundary—not a universal reasoning claim.
Benchmarks should expose their claim boundary
This is the operational value of the framework. Evaluation should record the transformations tested, the distribution shifts applied, the complexity range survived, and the model's uncertainty. The output is not a badge saying ‘reasoning achieved.’ It is an evidence envelope describing where the claim held and where it broke.
What remains unresolved
The deeper levels are harder to establish. Recovering a model's causal internal state is still an open technical problem. Social reasoning and embodiment sit outside the current framework, and phenomenal awareness is explicitly treated as untestable rather than smuggled into a benchmark result. Those limits strengthen the proposal: a useful standard should say what it cannot certify.
The assurance lesson
For Shrewd Research, the takeaway is simple: benchmark performance is an observation, while capability is a claim. The path between them must be explicit, adversarially tested, and narrow enough to falsify. The metre does not change when the object being measured becomes more impressive. AI evaluation needs the same discipline.