The promise of artificial intelligence, particularly the dazzling capabilities of large language models, often comes with a subtle, yet deeply unsettling caveat: inconsistency.
Imagine a doctor’s diagnosis varying wildly depending on when you asked, or a financial forecast shifting with each refresh, even with identical data.
Such capriciousness would be unthinkable in most critical fields.
Yet, in the burgeoning world of LLMs, we have, until now, largely accepted a curious paradox: the same prompt, fed into the same model, can yield entirely different outputs.
This isn’t merely a quirky characteristic of a “probabilistic” machine; it’s a fundamental erosion of reproducibility, reliability, and ultimately, the very trust we seek to place in our most advanced AI systems. For a deeper understanding of this need for reproducibility in AI, read more.
This frustrating enigma has long plagued developers and researchers, often dismissed with a shrug and a nod to the “stochastic” nature of AI.
But a groundbreaking intervention from the Thinking Machines Lab, detailed in their recent blog post “Defeating Nondeterminism in LLM Inference,” rips back this veil of abstraction.
Their work doesn’t just demystify why LLMs behave this way; it offers a tangible, bitwise-identical path to true reproducibility, transforming our understanding of these powerful tools.
For anyone invested in the future of AI, these insights are nothing short of indispensable.
Many in the AI community have intuitively attributed these output variations to the “sampling” process, or perhaps the inherent parallelism of modern GPUs. For more on the impact of nondeterminism in LLM inference, you can check various discussions.
Indeed, even when the ‘temperature’ parameter – which governs the randomness of an LLM’s output – is set to zero (theoretically making sampling deterministic by always picking the highest probability token), the issue persists.
Running inference on local hardware with popular open-source libraries like vLLM or SGLang still results in maddeningly inconsistent outputs.
A popular theory, the “concurrency + floating point” hypothesis, pointed to the non-associativity of floating-point arithmetic on GPUs, where (a + b) + c doesn’t always precisely equal a + (b + c) due to finite precision and rounding errors, especially when operations are executed in a nondeterministic order by concurrent processes. For an in-depth look at floating point arithmetic in GPUs, refer to official documentation.
While this hypothesis held a kernel of truth regarding floating-point behavior, Thinking Machines Lab’s research reveals it doesn’t tell the full story.
They demonstrated that running the same matrix multiplication on a GPU repeatedly will consistently provide bitwise identical results.
This crucial finding indicated that the core issue wasn’t simply floating-point math or concurrency in isolation.
The true culprit, they discovered, is far more subtle: it’s the lack of “batch invariance” in the low-level computational units, or “kernels,” used in the LLM’s forward pass, combined with the nondeterministic variation in batch sizes due to server load. For more on the significance of batch invariance in AI, review recent studies.
Batch invariance is a critical property where the computation results for a specific element in a batch should not be affected by other elements in that batch or by the total batch size.
Thinking Machines Lab empirically proved that many existing matrix multiplication implementations are not batch-invariant.
This means that if your individual query is processed as part of a batch of 10 requests, its output might subtly differ compared to if it were processed in a batch of 100, simply because the order and scale of floating-point additions change.
From a user’s perspective, the server’s load, and thus the batch size, is effectively nondeterministic, directly influencing their request’s output.
This isn’t just a GPU problem, either; it impacts LLM inference on CPUs and TPUs alike.
To achieve this elusive batch invariance, the Lab outlined specific, ingenious strategies for critical LLM operations. For more on deterministic AI applications, visit this resource.
For RMSNorm, it involves fixing the reduction order for each element regardless of batch size, typically through data-parallel approaches. For Matrix Multiplication, standard data-parallel methods are preferred, carefully avoiding techniques like “Split-K Matmul” that can introduce variability.
Attention, the most complex component, requires meticulous management of the KV cache and a “fixed split-size” strategy for FlashDecoding to ensure a consistent reduction order for each token, irrespective of how many other tokens are processed concurrently.
While some of these strategies might entail a slight performance trade-off in unoptimized scenarios – their example showed a 20% overhead compared to cuBLAS – this is a temporary cost for a monumental gain.
The practical demonstration of their batch-invariant-ops library, integrated with vLLM, was nothing short of striking.
When prompting a Qwen model for 1000 completions at temperature=0, the default setup yielded 80 unique outputs.
With Thinking Machines Lab’s batch-invariant kernels enabled, all 1000 completions were identical. This is the mathematical determinism we expect from a temperature=0 setting, finally achieved.
The implications of this breakthrough are profound, resonating across the entire AI ecosystem.
For scientific reproducibility and trust, this work is foundational. If researchers cannot consistently reproduce results, validating new models and building a reliable body of knowledge becomes a Sisyphean task. You can read more about transparency and reproducibility in AI.
More broadly, for the public and regulatory bodies, determinism is the bedrock of trust, assuring that an AI behaves predictably under identical conditions. In critical applications, from healthcare diagnostics to financial modeling and autonomous systems, predictable and consistent outputs are not just desirable, they are non-negotiable.
An AI that offers varying diagnostic probabilities for the same patient data, or different interpretations of the same legal contract, would be deemed unreliable and unsafe.
Batch invariance is the pathway to making LLMs truly reliable for high-stakes environments.
Furthermore, for developers, the ability to consistently reproduce bugs is a superpower. Nondeterminism transforms debugging into a frustrating game of “whack-a-mole.” Deterministic inference promises to streamline the development cycle, making complex LLM systems easier to build, test, and maintain. For additional challenges related to AI debugging, explore various resources.
Perhaps most significantly, for Reinforcement Learning (RL), nondeterministic inference implicitly turns “on-policy RL” into “off-policy RL” due to numerical divergences between training and inference.
Achieving bitwise identical results between the sampler and the trainer enables “true on-policy RL,” preventing training instabilities like reward collapse and opening new doors for more robust and effective AI agents.
This also extends to responsible AI and auditability, making it easier to trace, explain, and mitigate biases or unintended behaviors.
This research by Thinking Machines Lab represents a pivotal moment, moving us away from treating LLMs as inherently “probabilistic black boxes” and towards a more rigorous, principled engineering discipline. For insights on reliable AI systems, read further.
For too long, we’ve accepted numerical variations as an unavoidable consequence of parallel computing. This work forcefully rejects that “defeatism.” I predict that bitwise reproducibility will swiftly become a standard requirement in enterprise-grade AI deployments, particularly in heavily regulated industries.
While there may be initial performance overheads, these are temporary trade-offs. As these batch-invariant kernels are further optimized and integrated into core frameworks, the performance gap will undoubtedly narrow, making deterministic inference a practical reality for a wider range of applications.
This capability unlocks new frontiers for controlled experimentation, A/B testing, and robust validation of LLMs.
It empowers developers and researchers to reason about their models with greater confidence, transforming LLMs from fascinating but sometimes erratic entities into predictable, controllable, and ultimately more trustworthy tools.
The quest for predictable AI is not just an academic exercise; it’s a fundamental step towards building truly reliable, trustworthy, and impactful intelligent systems.
Thinking Machines Lab has provided a clear blueprint, reminding us that even in the most complex corners of machine learning, careful engineering and a refusal to accept “good enough” can lead to breakthroughs that redefine what’s possible.
Marketing buzzwords often obscure the true nutritional value of packaged goods, leading consumers to make choices that contradict their health goals.
A re-examination of Enlightenment philosophy reveals how the Chicago School narrowed the definition of freedom, shifting it from a social imperative to an isolated economic metric.
The Federal Trade Commission has launched a formal investigation into Anthropic and OpenAI, citing concerns over deceptive model behavior and public safety.
