HybridAgent Thinks in Latent Space and Acts in Text

An agent doesn’t need to write out every thought as text; it only needs text when it calls a tool or talks to a human. That’s the bet behind HybridAgent, from Stanford, UIUC, and Together AI. The model reasons by feeding its own hidden states back into itself, then drops into ordinary text whenever it needs Python, search, or a final answer. On Qwen3 backbones at 4B, 8B, and 14B, it beats matched text-only agents on accuracy while doing the work in far fewer decoding steps.

• 52.4 vs. 43.7 on GAIA (14B HybridAgent vs. the ARPO text-RL agent); 49.5 vs. 40.5 on WebWalker
• 35.5% to 54.7% fewer tokens and 1.8x to 2.6x faster inference than the baselines
• 2.7x to 3.7x cheaper: decoding cost per million questions drops from $470 to $176 at 14B, and $359 to $98 at 4B
• The latent thoughts matter: swapping them for another problem’s latents drops MATH500 from 88.2 to 48.6
• About 10M extra parameters (at 4B) for the adapter that makes it work

How one model runs in two modes

Latent reasoning methods like Coconut already showed a model can “think” by recycling its last-layer hidden state as the next input, skipping token decoding entirely. The catch is that pure latent reasoners can’t call tools and nobody can read what they’re doing. HybridAgent keeps both modes in a single generation stream.

The key piece is HybridLink, a two-layer residual adapter (roughly 2d² parameters) that projects last-layer hidden states into the input embedding space. The residual matters: a bare identity mapping scores 71.4 on MATH500, while the two-layer residual version scores 88.2. The model exits latent mode on its own when HybridLink’s output lands close (by cosine similarity) to the </latent> token embedding.

Training happens in two phases:

• HybridSFT (54K hybrid trajectories): first teach the text format, then train HybridLink with the backbone frozen, then a “self-conditioned” pass where the backbone learns to reason over latents produced by its own adapter. That last step is the big one. On Qwen3-14B-Base, it lifts AIME25 from 23.8 to 32.5 and MATH500 from 73.8 to 84.2, matching plain text SFT.
• HybridRL (12K reasoning + 1K agentic search examples): policy gradients apply only to sampled text tokens, with latent spans replayed exactly. The reward covers correctness, format, and using both tools. It never rewards mode choice directly, yet over training the 14B model shifts more of its generation into latent mode on its own.

Better scores, fewer decoding steps

At 14B, HybridAgent beats the same-size text-RL baseline and frontier-model agents built on Kimi-K3 and GPT-5.6-Luna on every agentic benchmark:

Benchmark	HybridAgent 14B	ARPO 14B	Kimi-K3 agent	GPT-5.6-Luna agent
GAIA	52.4	43.7	41.7	37.9
WebWalker	49.5	40.5	28.0	29.1
HLE	16.0	10.0	11.7	11.7
SealQA (Seal-0)	24.3	16.3	21.7	23.3

On math and knowledge tasks the margin is smaller but consistent: 14B HybridAgent beats ARPO on AIME26 (45.8 vs. 43.3), GPQA-Diamond (55.1 vs. 54.0), and MuSiQue (34.5 vs. 31.5).

The efficiency gains come from latent steps skipping the output head and sampling, and from shorter trajectories overall: up to 54% fewer total generation steps, with 43% to 49% fewer visible text tokens. The authors’ cost estimate charges every latent step as a full forward pass, so it doesn’t flatter the method:

Text mode still does its job. Valid tool calls execute successfully 99.7% of the time, the same as GRPO-trained text agents, and HybridAgent’s tool calls more often return the evidence that ends up in the answer (4.5 to 7.7 points higher). Hiding most reasoning also didn’t make the agent harder to audit: with only the visible text, a GPT-5.6-Luna judge located the first error in 99.2% of failed 8B rollouts, vs. 94.2% for a matched text agent.

Are the latent thoughts doing real work?

A fair worry with latent reasoning is that the hidden steps are just padding: extra compute with no meaningful content. The authors test this by tampering with the latents mid-generation and measuring MATH500:

Two things stand out. Content matters more than magnitude: swapping in another problem’s latents hurts far more than heavy random noise, so the model is reading what’s in those vectors. And each latent sits very close to a real token embedding (snapping it there costs only 1 point), which is what makes the latents readable. Mapping each step to its nearest token produces a pseudo-text trace you can inspect.

Also worth noting: forcing the trained model to reason only in text drops it to 78.8, below the plain GRPO text agent (87.4). Once trained this way, the model depends on its latent mode.

Caveats

• Not a clean sweep. At 8B, the ARPO baseline beats HybridAgent on HLE (18.9 vs. 14.4). The “+10.4%” headline is an average, and per-benchmark gains over the strongest text-RL baseline are often 1 to 3 points on math.
• Readability is a proxy. “Interpretable” here means latents lie close to token embeddings and their nearest-token readouts match the question’s vocabulary. That isn’t the same as a faithful explanation of the computation.
• You can’t serve this today. The authors note current inference engines don’t support multi-turn hybrid generation; they built custom training and inference support. The cost figures are estimates from FLOPs and an assumed 10% GPU utilization, not measured production bills.
• One model family, small scale. Every result is on Qwen3-Base at 4B to 14B, with HybridLink frozen during RL. Whether it holds at frontier scale is untested.
• The safety pitch cuts both ways. The authors argue latents help providers: there’s no reasoning text to leak, and only the provider’s HybridLink can read the trace. That also means users and third-party auditors see less of the agent’s reasoning than they do with today’s text agents.

For agent builders, the practical takeaway is about where the tokens go. Most of an agent’s text output is thinking that only the model reads. HybridAgent shows you can move that into hidden states, keep text for tool calls and key decisions, and cut decoding cost by roughly 3x without losing accuracy. If inference engines add support for the mode switching, this could become a standard post-training recipe rather than a research prototype.