ViBench 2.0

ViBench 2.0 is a long-horizon software-building benchmark for AI coding agents released by Replit on October 28, 2026. Instead of fixing a bug in an existing codebase, the agent starts from an empty project, builds a first version of a product from a short spec, then adds up to 18 features on top of its own code without knowing what comes next. It joins Fireworks' Specialized Intelligence Index (SII).

Announcement: Replit blog · Leaderboard: Fireworks SII · Launch post: Replit on X · Code: ViBench/vibench-public

Key points

1. What it measures: whether an agent can ship a whole working product, not just finish one coding task. Each of the 17 products is modeled on a real one such as Uber, GitHub or Google Docs, with 241 feature requests and 544 test plans in total. Eight products are public in the Harbor format; nine are held out.
2. Why it is hard: the agent must choose its schema and routes before seeing later features, so later features can break earlier ones. Tests check clean deployment on an empty database, existing data, invariants (split-fare shares must add up to the cent), stale state and concurrency.
3. Grading: after the last feature, a grading agent (Claude Opus 5.5) runs every test plan in a real browser by clicking and typing. Replit planted 34 bugs to check the grader, and GPT-6.1 Sol re-grading confirmed 99.2% of its passes.
4. Results (pass@1, mean of 4 builds per product): GPT-6.1 Sol 64.7% ($12 per build), GPT-6 Astra 60.3% ($90), Claude Opus 5.5 48.5%, Claude Sonnet 5.5 19.1%, Grok 4.7 1.5%. DeepSeek V4.1 Flash passes 77% of test plans for ~$2.50 per build, but no open-weight model passes a full build.
5. Lineage: ViBench 1.0 (ACM CAIS '26) was modeled on simpler apps like whiteboards and resume builders and introduced the "Vibe-on-Vibe" setting where models extend their own code. 2.0 stretches that into a full product evolution.

How to use it

• Read the leaderboard on Fireworks' SII page to compare models by pass rate, cost per build and build time.
• Run it yourself: the eight public products and harness are on GitHub; see vibench.ai for the paper.

Good fits

• Picking a model for app-building agents where later edits must not break earlier features
• Weighing cost vs reliability (Sol is far cheaper per build than Astra at similar pass rates)
• Researching long-horizon agent planning

Note

The official numbers come from Fireworks' runs on one harness and settings; your own agent setup may score differently.