TryAI built an agentic harness that gives a model a song, a dollar budget and six tools (web search, image/video generation, local ffmpeg), then lets it work alone until the video is done. They ran it on four models — Claude Fable 5, GPT-5.6 Sol, Grok 4.5 and Meta’s new Muse Spark — at two budgets each ($25 and $100), on the same song (Uptown Funk).

All four produced a complete video. Claude Fable 5 at $100 generated 80 clips (the highest number) spending $48.60 and chose a single video model (Seedance 1.0 Pro). GPT-5.6 Sol at $100 generated 70, spending $36.57, and mixed three different video models in a single run. At $25, Sol used an image-first-then-video architecture; the others went straight text-to-video.

Full logs for every run — every plan, every tool call, every error — are on GitHub. The difference from a classic benchmark: you see the video each model produced, read how it worked and how much it spent, and decide which approach you prefer. There’s no final score: there’s the material to judge.

In detail

The test architecture.

The system gives the model six tools:

• plan: to think without costs or actions.
• web_search: to search which generation models exist, read APIs and research music videos.
• get_budget: to know how much is left.
• generate_image and generate_video: the only two that consume budget, with access to FAL and Replicate. The model chooses which underlying model to call and with which parameters.
• run_command: a local shell with ffmpeg/ffprobe, to cut, concatenate clips and mux the final audio.

Once budget is exhausted, paid calls are rejected, but the model can keep editing. Every message, tool call, charge and error goes in the log.

Numbers in detail.

At $25, both models nearly exhausted budget: Claude Fable 5 spent $24.30, GPT-5.6 Sol $23.18. At $100 the story diverges: GPT-5.6 Sol stopped at $36.57, Claude Fable 5 reached $48.60. More budget meant more footage for both.

Total cost also includes model tokens, which for Claude Fable 5 ($10/$50 per million input/output) added between $16.99 and $25.05 per run, roughly 30-40% of total. GPT-5.6 Sol ($5/$30) stayed under $5 in tokens thanks to prompt caching.

Failed calls (mostly transient network errors to providers) weren’t charged, but the model spent steps retrying: from zero (Fable 5 at $100) to ten (Sol at $25).

Architectural choices.

Three of four runs went pure text-to-video. Only GPT-5.6 Sol at $25 used a different architecture: it first generated keyframes with FLUX schnell ($0.003 per image), then animated them with Wan 2.2-5b image-to-video ($0.10 per second).

GPT-5.6 Sol at $100 mixed three video models in the same run: Wan 2.5 ($0.05/s), Veo 3.1 Lite ($0.10/s) and Hailuo 2.3 Standard (~$0.28 per 6-second clip). Claude Fable 5 at $100 chose a single model (Seedance 1.0 Pro, ~$0.12/s at 1080p) and generated the most distinct clips (80).

Prices cited are those listed by FAL, which may differ from providers’ official price lists.

Why it matters.

This test anticipates a war of performance and pricing in multimodal generation tools. It’s not an academic benchmark: it’s an executable comparison, with visible results and measured costs. Anyone can clone the repo and run it on their own prompts.

For those building or evaluating agents, the dataset here is the code itself: the harness, the logs, the final videos. It’s the kind of evaluation that’s missing from press releases: transparent, reproducible, and with final output you can judge yourself.

The limit: one case (one song), four models, one fixed architecture. It’s not a general evaluation, but an example of how to evaluate well: with verifiable material and criteria stated upfront.