Together Compiler

Together Compiler (TKCC) is Together AI’s continual inference compilation system, announced 2026-11-04. Swarms of agents propose kernel/forward-pass changes while a persistent engine hot-reloads candidates—so optimization can re-run as models, traffic shapes, and chips drift, without cold-starting vLLM/SGLang-class engines each trial.

Together claims up to about 2.6× higher TPS versus strong open engine baselines in under 24 hours, plus InferenceX low-latency numbers such as 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash, already serving trillions of tokens on serverless endpoints. It is framed as the first shipping piece of Polymorphic Inference.

Problem

Open-model architectures and traffic change faster than hand-written kernels. Classic agent tuners stall on engine restarts. Together Compiler’s persistent engines raise experiment throughput (Together cites ~252×).

Differentiation

A production optimization loop on Together’s cloud—not a downloadable desktop compiler. Versus static megakernels: continual recompilation under traffic drift.

Limits

• Figures are mostly vendor-reported.
• Primarily a Together-side benefit; not clearly a standalone self-host product.
• Disambiguate from classic LLVM-style compilers by saying “Together Compiler”.

Who it’s for

Teams already serving open models on Together who care about TPS/cost under shifting traffic.

Get started

1. Read the Together Compiler post.
2. Check whether your Together endpoints already run continual optimization.
3. Replay production traces to measure latency/throughput deltas.