SpecPilot

SpecPilot is an autonomous algorithm-and-systems optimization loop from UC Berkeley, Princeton, Stanford, and NVIDIA, aimed at production speculative decoding stacks (project page, code, arXiv:2611.02501). A cost model finds gaps between a tuned stock system and its estimated ceiling; parallel coding agents explore fixes; token forcing checks every output token against stock so only lossless, significant gains are kept; validated mechanisms are stored in structured memory for reuse across deployments.

Across four target models, three drafters, two GPUs (H100 / RTX PRO 6000), and five benchmarks, SpecPilot lifts speedup over autoregressive decoding from stock’s 4.46× to 10.87× (single request) and from 1.94× to 3.93× with batching—no training and no quality loss. Lead author Kunjun Li announced it on 2026-11-04; Chinese X amplified the same day. Not to be confused with the unrelated girishr/SpecPilot SDD CLI for Claude Code/Cursor.

Problem

Parallel drafters (e.g. DFlash2, DSpark) can propose a whole block of tokens per pass, but released stock systems often leave much of that speed unused—and the right fix is deployment-specific (serving regime, drafter, target, GPU). Hand-tuning every stack does not scale.

Differentiation

• Agentic loop: cost model → parallel explorers → controller that accepts only significant gains confirmed on unseen requests.
• Lossless hard constraint: token forcing vs stock blocks reward hacking that speeds up at the cost of changed answers.
• Discovered reusable pieces including a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification, and several batching mechanisms.

Limits

• Reported speedups are author-paper numbers; reproduce on your own serving stack before capacity planning.
• The search agents need a strong coding model (paper uses Claude Opus 5.5 + Claude Code) and GPU budget.
• Easy to confuse with the unrelated SDD CLI of the same name—prefer StargazerX0/SpecPilot or the paper title in citations.

Who it is for

LLM inference / speculative-decoding engineers and teams studying agentic mlsys discovery.

Getting started

1. Read the project page and arXiv:2611.02501; note stock baselines and eval protocol.
2. Clone StargazerX0/SpecPilot and follow the repo to reproduce sealed evals on your GPU/model.
3. To reuse mechanisms only (12-bit format, batch verification, …), port matching pieces first, then let agents search inside a closed environment.