Consortium for Verified Computing

01 / The mechanism and its boundary

What is being described

Thinking Machines Lab describes itself as "an artificial intelligence research and product company" S-3562. Its work relevant here is Batch-invariant inference kernels (Thinking Machines):

In September 2025 it published batch-invariant kernels for LLM inference, arguing that varying batch sizes are the main reason LLM endpoints give nondeterministic outputs S-1009. Its MIT-licensed batch_invariant_ops library replaces several PyTorch operations with batch-invariant versions S-1813. See deterministic and bit-exact inference.
It reports that, with the kernels, 1,000 temperature-zero completions from Qwen3-235B were identical S-1009. SGLang built its deterministic mode on the kernels S-1012, and vLLM's developers state that its batch-invariant mode is based on the same work S-1814.

S-3562 / Tier B
Thinking Machines Lab ↗
· 2026 · Thinking Machines Lab
Supports: self-description
Locator: homepage
Version and catalogue details
S-1009 / Tier C
Defeating Nondeterminism in LLM Inference ↗
H. He, Thinking Machines Lab · 2025 · Thinking Machines Lab: Connectionism
Supports: batch size as the main cause of nondeterminism; batch-invariant kernels; Qwen3-235B result (provider-reported)
Version and catalogue details
S-1813 / Tier B
thinking-machines-lab/batch_invariant_ops (GitHub repository) ↗
Thinking Machines Lab · 2025 · GitHub
Supports: MIT-licensed batch_invariant_ops library
Locator: README
Version and catalogue details
S-1012 / Tier C
Towards Deterministic Inference in SGLang and Reproducible RL Training ↗
The SGLang Team · 2025 · LMSYS Org blog
Supports: SGLang's deterministic mode built on the kernels
Version and catalogue details
S-1814 / Tier C
[Feature]: Batch Invariant Feature and Performance Optimization (vLLM issue #27433) ↗
vLLM project contributors · 2025 · GitHub (vllm-project/vllm issues)
Supports: vLLM developers' statement that batch invariance is based on the post
Version and catalogue details