Prism Arena Is Not a Leaderboard

Prism Arena does not ask for an explanation of a 3D scene and stop at fluent prose. It gives each contender the same natural-language task pack and requires a SceneSpec v1 JSON artifact. That artifact must validate, execute in the shared runtime, render into inspectable frames, and survive a blind rubric review. The measured object is therefore not the response alone. It is the response moving through a delivery channel, a validator, a recorded-run selection rule, a renderer, a capture path, and a judge panel.

The completed corpus is explicit: eleven models multiplied by ten tasks multiplied by two recorded runs produced 220 runs. A recorded run could retry one failed OMP invocation; that internal retry count was not persisted, so reliability is measured per recorded run rather than per raw provider call. The export records 203 valid and 17 invalid runs. Four judges scored the selected run in each of the 110 model-task cells, producing 440 judgments. All 110 cells are fully judged. Those counts establish coverage. They do not turn the resulting order into a claim about every task a model might perform.

This distinction matters because the harness failed in ways that looked like model behavior. A provider route could answer a toy probe and reject the real task. A tool-using model could deliver the right artifact to a file while the response channel contained only a summary. A capture process could show a plausible scene from the wrong recorded run. None of those failures lives inside the model weights, yet each can change a score unless the benchmark treats the surrounding system as part of the instrument.

SceneSpec v1 is the common contract. Before a submission reaches a judge, the schema checks geometry, materials, lights, an ordered timeline of two to eight phases, and the span arithmetic that keeps each action inside its phase. A response that describes the right scene but omits a required field is not rescued by intent. It is invalid. In this benchmark, validity is a floor rather than a scored dimension.

Every valid artifact then enters the same React Three Fiber runtime, with Anime.js as the only motion system. That shared execution path removes a large source of implementation variance: contenders do not bring their own renderer, shaders, physics engine, or animation library. The comparison asks what each model can express through the same constrained scene language and what that language produces when executed by the same code.

The shared runtime also defines the boundary of the claim. Prism Arena measures constructive spatial, material, and temporal competence inside SceneSpec and this renderer. It does not measure unrestricted 3D development. The headline score is the panel-mean total across ten tasks, each scored out of twenty. The selector sends the earliest valid recorded run in each cell to the panel; if neither run validates, it sends the earliest invalid run instead of assigning an automatic zero. In the sole double-invalid cell, all four judges gave the selected artifact zero. The separate per-recorded-run validity rate preserves the reliability information that best-of-two selection by validity would otherwise hide.

The first channel confound was simple. When a write tool was available, agentic models wrote the requested SceneSpec to a file and returned a prose summary in the response. A response-only parser then marked that run malformed. The requested artifact existed, but the harness had looked in the wrong place. Penalizing that behavior would have measured compatibility with one return channel, not scene construction.

The correction was a uniform channel: every contender ran with tools, skills, and rules disabled. That choice does not reveal a model's maximum agentic performance. It isolates the capability this corpus intended to compare: producing the complete SceneSpec in the response consumed by the validator. A different benchmark could allow files and tools, but it would need to collect and validate those artifacts for every contender under the same contract.

Provider access created a second channel failure. One route passed a one-line health check and then returned `Model not found` on every real generation. The only useful smoke test sent the actual task and validated the actual contract. Transport faults were then classified as retryable provider failures rather than published model zeros. Route health, tool access, system instructions, and artifact collection are experimental variables even when they look like infrastructure.

Judges normally receive the SceneSpec and two rendered frames per contender. In an early pipeline, frame capture drove the interactive arena, which displayed the newest recorded run, while judging selected the earliest valid run or, if neither validated, the earliest invalid run. The rules were individually reasonable and collectively incompatible. Capture and scoring disagreed in 100 of 110 model-task cells.

Nothing crashed. The frames looked legitimate because they came from real submissions, but they did not always depict the artifact under review. Every component succeeded locally while the combined system broke artifact identity. A polished screenshot cannot serve as evidence unless it carries provenance back to the exact recorded run selected for scoring.

Capture and judging now derive the recorded run from one shared selection rule. The selected run is pinned in the arena URL, and its identifier is stored with the captured frame so stale evidence cannot be silently reused. Frames are all-or-nothing within a judging batch: if any contender is missing its full configured frame set, the batch falls back to JSON-only rather than giving some contenders visual evidence and others none.

The panel has four judges from three model families; every judge is also on the contender roster. Contender names are blinded. Three judges receive an explicit high reasoning-effort setting; Gemini 3.6 Flash carries effort in its route name and accepts no separate flag. The three-family panel broadens the leave-one-out baseline but does not remove self-preference. Blinding does not create an independent ground truth or perfectly matched judging conditions.

The clearest warning came from judge redundancy. On a subset of 59 selected runs, the two same-family judges had a residual correlation of 0.502, compared with 0.114 for a cross-family control. That 4.4× gap looked like a reason to drop one of the same-family judges. On the complete corpus of 110 selected runs, the same comparison was 0.559 versus 0.520, a difference of 0.039 and no basis for removing a judge. The panel stayed at four. The partial result was useful for monitoring, not for redesigning the experiment.

Self-preference needs the same control. Raw family means suggested that the two Anthropic judges scored Anthropic contenders 0.9 and 0.7 points above others, but that comparison mixes judge behavior with contender quality. Within-contender own-model residuals were −0.37 for Claude Opus 5, −1.10 for GPT-5.6 Sol, +0.50 for Claude Fable 5, and −0.43 for Gemini 3.6 Flash. They ranged from −1.10 to +0.50 and changed sign across judges. These own-model residuals are narrower than an own-family comparison, so they do not show that self-preference is absent. They show why the raw family means are not enough.

The residual baseline is made from other judges that are themselves not family-neutral. The corpus has only two recorded runs per cell, the panel has no human calibration set, and effort cannot be configured identically across providers. These limits belong beside the scores.

The full-corpus ranking is more defensible than the intermediate one because all 110 expected model-task cells have recorded runs and all 110 are fully judged under one harness version and one judge version. It is still provisional. Ten tasks sample a narrow form of constrained scene-and-motion synthesis, and two recorded runs per cell are too thin for confidence intervals on the ordering. We make no claim of statistical significance.

Read a headline score as shorthand for a longer sentence: under this response-only channel, this model had its selected artifacts from these ten SceneSpec tasks rendered by this React Three Fiber and Anime.js runtime and scored by these four judges with this rubric in July 2026. Remove any condition and the number starts to describe a different experiment.

The bounded result can identify artifacts worth inspecting, tasks that separate models, schema failures that deserve a retry policy, and judge disagreements that need human review. It cannot establish that one model is universally better at coding, spatial reasoning, design, or agentic work. The next version should raise recorded runs from two to at least five, report reliability by rubric dimension, add a stratified human-rater study, and model task, run, and judge variation instead of printing a strict order when uncertainty overlaps.