Cola2

Cola2 is ByteDance Seed's continuous-latent language model: it co-adapts semantic compression and discrete text generation — a dynamic-compression VAE encodes spans into compact continuous latents, an ~4B block-causal DiT generates latent blocks via Flow Matching, and a span-causal token decoder realizes discrete tokens with latent feedback so continuous and discrete histories stay aligned. Weights: https://huggingface.co/ByteDance-Seed/Cola2 ; code: https://github.com/ByteDance-Seed/Cola2 ; paper arXiv 2610.31305; project page https://hongcanguo.github.io/Cola2/ .

Ship notes (official HF, 2026-11-06):

• ~9B-param base model (not instruction-tuned chat);
• Trained on 346.03B source tokens; reported 60.10 eight-task average (mean over latent-noise samples);
• Checkpoint is a research / still-unconverged DiT release, with a comparison table vs Llama-3-8B and peers;
• Inference needs trust_remote_code=True; default paper preset includes 8-step Euler DiT, DiT CFG 10, decoder CFG -0.9, etc.;
• Repo code Apache-2.0; weights and third-party assets follow NOTICE.

Why it matters

Cola2 ships continuous semantic compression + diffusion-style latent generation + discrete decoding as a same-day HF/GitHub product name from Seed — a concrete non-pure-AR line.

Method in one line

Stage I jointly trains encoder and token decoder; Stage II keeps adapting the VAE while a DiT learns the detached latent distribution; at inference it loops propose-latent → decode-tokens → re-encode under the fixed span layout.

Limits

• Latent generation is stochastic — seed a torch.Generator for comparisons;
• Unconverged checkpoint; eight-task average still trails well-trained 7–8B AR/LLaDA peers;
• Base model only — not a chat assistant out of the box.

Who it's for

• Researchers of continuous-latent LMs and DiT/Flow Matching text generation;
• Teams reproducing Seed Cola2 inference and the eight-task smoke bench;
• People comparing AR / LLaDA / Cola-MDM-style paradigms.

Getting started

1. hf download ByteDance-Seed/Cola2 and verify SHA256SUMS;
2. Load with AutoModel.from_pretrained(..., trust_remote_code=True) per the card;
3. Clone https://github.com/ByteDance-Seed/Cola2 for scripts/run_chat.sh / benchmarks;
4. When citing scores, note three-seed means and the unconverged checkpoint.