Same Model, Same Seed, Two Answers

Local inference determinism check · measured 31 July 2026 · runtime 65 seconds

The claim under test

With a pinned local backend, greedy decoding is bitwise reproducible — and changing the compute backend, changing nothing else, can change the answer.

Run specification

prompt    List three properties of floating point addition.
model     Qwen2.5-1.5B-Instruct, Q4_K_M GGUF
runtime   llama.cpp b10210
hardware  Apple M2 Pro, 32 GB, macOS 26.5.2
fixed     --temp 0   -s 42   -n 80
varied    -t   ∈ {2, 4, 8, 12}     (CPU threads)
          -ngl ∈ {0, 12, 99}       (layers offloaded to GPU)
runs      14

Result

Fourteen runs produced exactly two byte-identical outputs. Thread count made no difference at all — 2, 4, 8, 12, same hash on each backend. Repeated runs on the same backend: same hash. The only variable that changed the answer was the compute backend itself.

The two answers

Neither is right, and they are wrong in different ways

The CPU answer is flatly incorrect. You cannot add mantissas and exponents separately: addition requires aligning exponents first, then adding, normalizing, and rounding. Adding exponents is what multiplication does.

The GPU answer is true in a trivial sense and misleading as stated. The result is a representable value — but it can be infinity on overflow, negative zero, or NaN, and it is almost always rounded rather than exact. Presenting closure as a headline property under “Additivity” implies a clean algebraic behaviour that floating point addition specifically does not have. The interesting facts — that it is not exact and not associative — are omitted in both answers.

If one backend were reliably correct you could just pick it. What you actually get is two plausible, confidently-worded, differently-defective answers — and nothing in either one signals which kind you got. The defect is not in either answer. It is that neither carries a quality flag.

Why this matters for hosted inference

Locally, pinned, each backend held still — which means you can hold a baseline, change one thing, and know the difference came from your change. A hosted API cannot promise even that: Thinking Machines (2025) locates the cause in batch-invariance — kernels pick reduction strategies based on batch shape, so output can depend on who else was in the batch. Backend divergence in the exact tool used here is also long documented in llama.cpp issue #2838, with follow-on academic work in LLM-42 (2026). This report confirms a known kernel-level result on a laptop in 65 seconds; the contribution is connecting it to what infrastructure should do about it.

What Symbia does about it

Not pretend the model is deterministic. Every reply in the Symbia stack carries its epistemic status: which lane it travelled (canonical, conditional, apocryphal), what decided it (method: declaration | model), and whether it is recomputable. Decidable work is routed to declarations that cannot flap; model output is labelled as model output, with the execution environment it depends on. See lanes, claims, and lineage and deterministic routing — results.

Postscript: we asked the running stack

On 13 August 2026 the same prompt was sent to a running Symbia stack, through the messaging service, as an ordinary conversation message. The coordinator routed it; the calculator answered:

The reply's provenance envelope records: arena RETRIEVED (authored text from the rule “Invalid Input” — not a computed value); every routing step with its output digest; the coordinator→calculator delegation with its hash; an ed25519 signature; and a basis sentence stating the routing choice is reproducible from the message. Where two model backends confidently produced different wrong answers, the stack produced a signed, honest statement of its own limits — and a suggestion, which the section above takes. The unedited envelope: determinism-check-live-reply.json.

Reproduce it

One script, one small download, about a minute: determinism-check-reproduce.sh. Any prompt with a checkable answer serves; that is the point of shipping the script.

Follow-up

The suggestion in the postscript was taken on 13 August 2026: Fourteen Identical Answers, All Wrong reruns the experiment with a decidable, non-arithmetic task. The model produced fourteen byte-identical wrong answers; a sorter assistant registered through the platform API computed the correct one, signed. Determinism is not correctness.

Correction note

An earlier draft of this analysis called the GPU answer correct and the CPU answer wrong. Review showed both are defective — one badly, one subtly. In a report arguing that claims outrun evidence, the visible correction is worth more than the clean version.