Can a coding agent review sports film? What the benchmark measured

Does reasoning effort matter for video review?

Answer: Yes, a lot. Changing only the effort moved Sonnet 5.5 from 0.419 (default effort, n=2, range 0.413–0.426) to 0.931 (effort high, n=4, range 0.856–0.994) above chance. At effort max it gave no answer before the time limit.

Same model, svc-bench-3 drawn from a new seed for every run, frames track (stills only, no code); only the effort setting changed.

Board rows: What effort changed · svc-bench-3 (fresh seeds) · Transcripts 1 2 3 4 5 6 7 8 9 10 11 12 13

Bound: 13 runs in all, each on its own seed; 3 of 7 settings ran once. A measured difference, not a variance estimate.

Does a raw score tell you how well a model sees?

Answer: Not by itself. A blind guess that sees no video already totals 0.170–0.379, so read the above-chance column: Haiku 5.5 totals 0.440 but is only 0.198 above chance.

Blind guess total:

• svc-bench-3: 0.331
• svc-bench-2: 0.356
• svc-bench-1: 0.324

The blind guess copies the example answers in the task text. On fresh seeds each run has its own floor; those 15 floors total 0.170–0.379.

Above chance: 0 = no better than a blind guess that sees no video, 1 = perfect (per task, (score - floor) / (1 - floor)).

Board rows: svc-bench-3 (fresh seeds) · Before you trust this

Bound: Floors are computed by the tool for every suite and seed; one blind guess per suite, no randomness.

Can a coding agent count reps from video?

Answer: On these drawn drills, yes, when it may run code: 5 of 9 code-track runs scored 1.0 and none totalled below 0.000. On the frames track, by eye from stills, contenders spread from 0.000 to 1.000 above chance.

Code track: the agent writes and runs its own tracker. The drawn videos have flat colours, so a pixel tracker finds the ball; that is why the frames track exists.

Board rows: svc-bench-3 · svc-bench-2 · svc-bench-1 · svc-bench-3 (fresh seeds)

Bound: 9 code-track runs over 3 suites; the frames range is the contenders' mean above chance on the fresh-seed board.

Is a small local vision model good enough?

Answer: Not for this: qwen3-vl:4b-instruct-q4_K_M scored 0.000 above chance on every board it ran: totals of 0.125 and 0.077 against blind guesses of 0.331 and 0.232.

Run through Ollama on the runner's own GPU, one request per task with that task's contact sheets.

Board rows: svc-bench-3 · svc-bench-3 (fresh seeds) · Transcripts 1 2

Bound: 2 runs of one quantised model at temperature 0; a larger local model may do better.

What does a near-perfect review cost?

Answer: On the fresh-seed board, of the contenders at 0.99 or more above chance with a recorded cost, the cheapest was Opus 5.5: $2.55 and 347 s per run (2 runs), scoring 0.997 above chance. gpt-6-astra, effort xhigh also reached 1.000 but reports no cost.

Cost is what the agent itself reported (Claude Code does, Codex and Ollama do not); time is wall seconds.

Not run on fresh seeds: Fable 5.1 on the fixed svc-bench-3 board: 0.494 above chance over 2 runs (best 0.988), $19.23 (1 of 2) per run.

Board rows: svc-bench-3 (fresh seeds) · svc-bench-3

Bound: Means over 15 runs; most contenders ran once or twice. Prices are per run of four short videos.

Where does each model fail?

Answer: The hardest task was stroke-rate: a mean score of 0.700 over the 13 runs that answered. (a rowing stroke rate, with a far boat at a close rate)

Mean recomputed score per task (0 to 1) on the fresh-seed board; the board's fold lists every answer and what was missed.

• juggle-touches: count one player's touches while a second player juggles next to them
• far-hoop-shots: made or missed at a 16 px wide far rim
• pass-releases: when each pass leaves the hand, with pass fakes
• stroke-rate: a rowing stroke rate, with a far boat at a close rate

Board rows: svc-bench-3 (fresh seeds)

Bound: Scores recomputed from each run's stored answer and its seed's truth; 15 runs.