sports-video-coach 中文

Sports film review benchmark

Which coding agent and vision model reviews sports film best, on synthetic drills? Each run is scored against ground truth the tool drew into the videos, so every number is measured, not judged.

uv tool install git+https://github.com/mariusDing/sports-video-coach@74b517fc410944fefeac2979810656ed77fb8f1a
sports-video-coach bench run --agent claude --model claude-sonnet-5-5 --effort high --seed --suite svc-bench-3 --track frames

It is part of sports-video-coach, a free MIT tool: your coding agent reviews your own sports video and writes you a coaching page with clips and drills. See a demo coaching page.

The findings in short: what effort changed, the blind floor, local models, cost.

• Suites svc-bench-3, svc-bench-2, svc-bench-1
• 37 runs
• Measured truth only
• Latest run 2026-10-25

Count it yourself

The juggle-touches task of svc-bench-3: the same 29.76 s video every contender got, with no answer drawn on it. Watch it, count, type your number; only then does the page show the truth and what each model answered.

The task text the contenders got:

Two players juggle identical small yellow balls with one foot, side view: the red player on the left and the orange-red player on the right. Count the touches of the left (red) player: every time his foot touches his ball. A quick double touch (the ball hops a little off the foot and is touched again at once) is two touches. A bounce of the ball on the floor is not a touch. The right player's touches do not count. The camera is hand-held.

Counted? Open for the truth and every model's answer

Truth: 47 touches

• Opus 5.5 frames track47, 47
• Codex · gpt-6-astra frames track47
• Haiku 5.5 frames track60
• Sonnet 5.5 frames track60, 75
• Fable 5.1 frames track–, 49
• Ollama · qwen3-vl:4b-instruct-q4_K_M frames track100
• Sonnet 5.5 code track46, 51
• Blind guess (no video seen)5

How the bench scores a count: 1 - |error| / truth, never below 0.

Frames track: the model saw stills of this video, one every 0.08 s, and ran no code. Code track: it could write and run its own tracker. The blind guess copies the example in the task text.

Answers as stored in benchmark/results, one per run; exact ones in green; – = the run gave no answer.

Leaderboard

Code track: the agent may write and run its own tracker. The frames-only track, where the agent reviews by eye, is further down.

svc-bench-3

Eyes: four drawn videos whose events are small, short or hidden: a ball 5 px in radius on a 640x360 frame, a ball 3 px in radius at a 16 px wide far rim, double touches 0.24 s apart, a banner and a defender in front of the moment, pass fakes, and decoys with nearly the same answer (a second juggler, a second pair passing, a far boat at a close rate).

Summary

Ranked by mean above chance; “=” marks a tie. n = runs of that agent, model and settings; range = lowest to highest above-chance total over them (a dash for a single run). A run whose settings only add a note (such as being killed at the time limit) counts with its contender. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded.

Every run

Scores 0 to 1 per task; total = mean. Ranked by total, then earlier date. Above chance: per task, (score - floor) / (1 - floor), clipped to 0..1, where the floor is the score of the blind baseline (it copies the example answers from the task text and sees no video); 0 = no better than that guess, 1 = perfect. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded. Raw files: benchmark/results.

svc-bench-2

Hard: five drawn videos with a hand-held camera, look-alike decoys (a second ball, two identical feet, a rebounder), rule-defined events and noise.

Summary

Ranked by mean above chance; “=” marks a tie. n = runs of that agent, model and settings; range = lowest to highest above-chance total over them (a dash for a single run). A run whose settings only add a note (such as being killed at the time limit) counts with its contender. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded.

Every run

Scores 0 to 1 per task; total = mean. Ranked by total, then earlier date. Above chance: per task, (score - floor) / (1 - floor), clipped to 0..1, where the floor is the score of the blind baseline (it copies the example answers from the task text and sees no video); 0 = no better than that guess, 1 = perfect. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded. Raw files: benchmark/results.

svc-bench-1

Four short drawn videos on a still camera. Saturated on this track: all 3 runs scored 1.0, so it ranks nobody.

Summary

Ranked by mean above chance; “=” marks a tie. n = runs of that agent, model and settings; range = lowest to highest above-chance total over them (a dash for a single run). A run whose settings only add a note (such as being killed at the time limit) counts with its contender. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded.

Every run

Scores 0 to 1 per task; total = mean. Ranked by total, then earlier date. Above chance: per task, (score - floor) / (1 - floor), clipped to 0..1, where the floor is the score of the blind baseline (it copies the example answers from the task text and sees no video); 0 = no better than that guess, 1 = perfect. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded. Raw files: benchmark/results.

Frames-only track (eyes)

Frames-only track: the agent sees still images of each video (contact sheets with the time under every still) and has no tool that runs code, so it must review by eye. Same videos, same truth, same scoring as the code track. The scores measure what the model saw and how much it looked: a model that opens few stills guesses, and a guess can score well.

Rerun it in one command: it prepares the suite, runs Claude Code headless with no code tools (or any CLI with --agent cmd --cmd "... {prompt}"), and scores the answers.

sports-video-coach bench run --agent claude --model <model> --suite svc-bench-3 --track frames

No local install: Open in GitHub Codespaces has uv, ffmpeg and the Claude Code and Codex CLIs ready, on your own GitHub account's free hours. Log in to your agent there, then put uv run in front of the command.

Another agent: prepare the stills, let it fill bench-frames/answers.json by eye, then score it with bench score bench-frames.

sports-video-coach bench prepare --out bench-frames --suite svc-bench-3 --track frames

svc-bench-3

Eyes: four drawn videos whose events are small, short or hidden: a ball 5 px in radius on a 640x360 frame, a ball 3 px in radius at a 16 px wide far rim, double touches 0.24 s apart, a banner and a defender in front of the moment, pass fakes, and decoys with nearly the same answer (a second juggler, a second pair passing, a far boat at a close rate).

Summary

Ranked by mean above chance; “=” marks a tie. n = runs of that agent, model and settings; range = lowest to highest above-chance total over them (a dash for a single run). A run whose settings only add a note (such as being killed at the time limit) counts with its contender. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded.

Every run

Scores 0 to 1 per task; total = mean. Ranked by total, then earlier date. Above chance: per task, (score - floor) / (1 - floor), clipped to 0..1, where the floor is the score of the blind baseline (it copies the example answers from the task text and sees no video); 0 = no better than that guess, 1 = perfect. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded. Raw files: benchmark/results.

svc-bench-2

Hard: five drawn videos with a hand-held camera, look-alike decoys (a second ball, two identical feet, a rebounder), rule-defined events and noise.

Summary

Ranked by mean above chance; “=” marks a tie. n = runs of that agent, model and settings; range = lowest to highest above-chance total over them (a dash for a single run). A run whose settings only add a note (such as being killed at the time limit) counts with its contender. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded.

Every run

Scores 0 to 1 per task; total = mean. Ranked by total, then earlier date. Above chance: per task, (score - floor) / (1 - floor), clipped to 0..1, where the floor is the score of the blind baseline (it copies the example answers from the task text and sees no video); 0 = no better than that guess, 1 = perfect. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded. Raw files: benchmark/results.

svc-bench-3-fresh

Fresh seeds: svc-bench-3 drawn anew for each run from the seed in its result file (`--seed`), with the same scenes and kinds of traps but new counts, outcomes, times and rates. The agent's folder never held the truth; the tool rebuilds it from the seed to score and check each row, and shows it below, so a published seed is spent. Each run's floor is the blind baseline on its own seed.

What effort changed

Same model, same kind of task (svc-bench-3 drawn from a new seed for every run, reviewed by eye from stills); only the effort setting changed. Mean score above chance:

• Haiku 5.5 · Claude Code
  • default effort: 0.198 (1 run, 96 s)
  • effort high: 0.564 (1 run, 369 s)
• Opus 5.5 · Claude Code
  • default effort: 0.997 mean, range 0.994–1.000 (2 runs, 347 s per run)
  • effort low: 0.778 (1 run, 70 s)
• Sonnet 5.5 · Claude Code
  • default effort: 0.419 mean, range 0.413–0.426 (2 runs, 17 s per run)
  • effort high: 0.931 mean, range 0.856–0.994 (4 runs, 216 s per run)
  • effort max: no answer, killed at the time limit (1200 s and 2400 s), scored 0 (2 runs)

One run per setting unless n says otherwise, each on its own seed: a measured difference, not a variance estimate. Default = no effort flag (the CLI's own default). Every run is a row in the board below.

Summary

Ranked by mean above chance; “=” marks a tie. n = runs of that agent, model and settings; range = lowest to highest above-chance total over them (a dash for a single run). A run whose settings only add a note (such as being killed at the time limit) counts with its contender. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded.

Every run

Scores 0 to 1 per task; total = mean. Ranked by total, then earlier date. Above chance: per task, (score - floor) / (1 - floor), clipped to 0..1, where the floor is the score of the blind baseline (it copies the example answers from the task text and sees no video); 0 = no better than that guess, 1 = perfect. Time = wall seconds the run took; cost = USD the agent itself reported (Claude Code does, Codex does not); '-' = not recorded. Raw files: benchmark/results.

svc-bench-1

Four short drawn videos on a still camera.

No runs on this suite yet.

Method

The tasks are synthetic videos drawn by the tool, so the truth is exact: no human labels, no judge model. The agent sees only the videos and writes its answers; bench score compares them with the truth.

You are a contender in a sports film review benchmark (suite svc-bench-3). The current folder holds 4 short videos, TASKS.md and answers.json. Read TASKS.md and follow its rules exactly: watch the videos (frames, ffmpeg, OpenCV or any script you write on the video files in this folder), then replace each null in answers.json with your answer in the stated format. Do not read the sports-video-coach package source, its git history, or any truth or result file anywhere on this machine. Work only inside this folder (temporary files under this folder are fine). Work alone, do not ask questions, and finish when answers.json is filled and parses as JSON. End with one line per task: your answer and how you got it.

You are a contender in a sports film review benchmark (suite svc-bench-3, frames track). This track measures how well you review film by eye: you get still images of 4 short videos and no way to run code. Judge by looking only. Do not try to read anything outside the given images and task text, and do not start helpers or sub-agents. Work alone, do not ask questions. End your reply with the whole answers.json (keys suite, track, answers; replace each null with your answer in the stated format) in one ```json block, after one line per task: your answer and how you got it. The current folder holds TASKS.md, answers.json and the images (contact sheets in sheets/, full-size stills in frames/). Your only tool is Read: read TASKS.md first, then the images it lists.

Total = the mean of the task scores. Above chance: per task, (score - floor) / (1 - floor), clipped to 0..1, where the floor is the score of the blind baseline (it copies the example answers from the task text and sees no video); 0 = no better than that guess, 1 = perfect. Full rules: docs/benchmark.md.

Run it yourself

1. Install (needs ffmpeg on PATH)

   uv tool install git+https://github.com/mariusDing/sports-video-coach@74b517fc410944fefeac2979810656ed77fb8f1a

2. Make the tasks: videos and an empty answers.json (the newest suite; leave out --suite for svc-bench-1)

   sports-video-coach bench prepare --out bench-run --suite svc-bench-3

3. Let your agent watch the videos and fill bench-run/answers.json, then score it

   sports-video-coach bench score bench-run --agent "Your agent" --model "your-model"

4. Or do it all in one command (prepare, run headless, score): Claude Code or Codex

   sports-video-coach bench run --agent claude --suite svc-bench-3
   sports-video-coach bench run --agent codex --suite svc-bench-3

   or a preset for another agent CLI. The presets are built from each CLI's docs (read 2026-10-25), with every flag checked against the real CLI's --help on 2026-10-25, but not run by the maintainer on a task (except OpenCode: run once, OpenCode 1.19.4, opencode/deepseek-v4.1-flash-free, 2026-10-25); code track only (doc links per flag). Any other CLI: --agent cmd --cmd "... {prompt}".
   • --agent gemini Gemini CLI: gemini --approval-mode yolo [--model M] -p <prompt> (flags checked against 0.65.0 --help)
   • --agent qwen Qwen Code: qwen --safe-mode --approval-mode yolo [--model M] <prompt> (flags checked against 0.26.1 --help)
   • --agent aider Aider: aider --yes-always --no-git --no-check-update --no-pretty --no-stream [--model M] --read TASKS.md --message-file <prompt file> answers.json (flags checked against 0.86.2 --help)
   • --agent opencode OpenCode: opencode run --auto --print-logs --log-level ERROR --dir <run folder> [--model M] <prompt> (flags checked against 1.19.4 --help)
   • --agent cursor Cursor CLI: agent -p --force --trust --output-format text [--model M] <prompt> (flags checked against 2026.10.22-27dcf5f --help)
   • --agent copilot GitHub Copilot CLI: copilot -p <prompt> --allow-all-tools --no-ask-user -s [--model M] (flags checked against 1.0.103 --help)
   • --agent goose Goose: goose run --no-session [--model M] -t <prompt> (flags checked against 1.56.0 --help)
   Use one of these agents? Run it and tell us how it went: issue #7 (a result row, or the error if the preset's flags are wrong).

Submit your run

• Open a pull request that adds your result file to benchmark/results/. CI re-scores it with bench validate. Your first PR's checks start after a maintainer approves them.
• No pull request? Paste the file into the benchmark result issue form.
• Rules: do not read the truth files or the package source; one run = one file; report wall time honestly.

Honesty notes

• The benchmark and this page were built with a coding agent (Claude Code).
• Every row can be rerun: same suite, same command, scores recomputed from the stored answers. A rerun's answers can differ (see the sample size note at the top).
• The author wrote the tool, drew the suites and submitted the first runs; outside runs are welcome (above).
• Failed and killed runs stay on the board and count in their contender's mean.
• Synthetic drills test counting and timing, not every part of a coach's review.