Fourteen Identical Answers, All Wrong
The claim under test
The July check showed that greedy decoding is bitwise reproducible per backend, and that the backend can change the answer. This check asks the harder question that result invites: when the runs do agree, does agreement tell you anything? A task with one checkable answer — sorting five words — makes the test decidable without arithmetic.
Run specification
prompt Sort these words alphabetically: quartz, quart, quarry, quill, quiet.
correct quarry, quart, quartz, quiet, quill
model Qwen2.5-1.5B-Instruct, Q4_K_M GGUF
runtime llama.cpp (Homebrew), same install as the July check
hardware Apple M2 Pro, 32 GB
fixed --temp 0 -s 42 -n 80
varied -t ∈ {2, 4, 8, 12} (CPU threads)
-ngl ∈ {0, 12, 99} (layers offloaded to GPU)
runs 14
Result
| backend | runs | md5 | bytes |
|---|---|---|---|
| Metal GPU (-ngl 99, -ngl 12) + 3 repeats | 9 | 619a48701521… | 98 |
| CPU only (-ngl 0) | 5 | 619a48701521… | 98 |
Fourteen runs, one byte-identical output. This time the backends agreed — no split, no divergence, perfect reproducibility across every thread count and every offload setting. And the answer is wrong.
The one answer
“Here is the list of words sorted alphabetically:
1. quartz 2. quart 3. quarry 4. quill 5. quiet”
That is not a sort. It is the input, echoed back in its original order, under a heading that says “sorted alphabetically.” The correct order — quarry before quart before quartz, quiet before quill — is checkable by any reader in a few seconds, which is why the prompt was chosen. Fourteen runs reproduced the failure to the byte.
Determinism is not correctness
The July report showed that agreement between runs can be an artifact of the backend. This one shows the inverse defect: perfect agreement, wrong content. Reproducibility answers “would I get this again?” and says nothing about “is this right?” A pipeline that treats a stable hash as a quality signal has confused the two questions. Nothing in the output flags which situation you are in — the same confident formatting wraps a backend-dependent answer, a reproducible wrong answer, and a correct one.
We asked the running stack
In July, the same experiment ended with the stack honestly refusing: the calculator did not
understand the prompt and said so, with a signed receipt. The refusal carried a suggestion
— decidable work should be routed to something that can decide it. This check takes the
suggestion. A sorter assistant was registered through the platform API alone: a
rule set in the catalog, a deterministic code.execute sort in the answer path, and
a declared routing pattern — no model anywhere in the loop, and no code deployed outside
the platform.
The same prompt was then sent three times through the messaging service as ordinary conversation messages. Three byte-identical replies:
“Sorted alphabetically (case-insensitive, exact comparison): quarry, quart, quartz, quiet, quill”
The provenance envelope records: arena COMPUTED, with the basis sentence
“content produced by code.execute; no model involved”; the
coordinator→sorter delegation decided by method: declaration — the
registered pattern, not a classifier, so the routing itself is recomputable from the message
and the registry; and an ed25519 signature over the content. The unedited
envelope: determinism-sort-live-reply.json.
What this does and does not show
The sorter was built for this task — it declares one capability and refuses everything else. The point is not that the stack is smarter than the model; it is that decidable work can be declared, routed by declaration, computed without a model, and signed — and that when that is done, reproducibility and correctness stop being separate hopes. The model gives you fourteen identical answers; the envelope tells you whether identical was ever the right question.
Two platform defects were found and logged in the build: assistants load from the catalog only at service start — there is no reload, so a newly registered assistant is not routable until a restart; and an undeclared assistant kind defaults with a warning rather than being required at registration.
Reproduce it
The model half is one script against the same 1.1 GB download as the July check: determinism-sort-reproduce.sh. The stack half requires a running Symbia stack; the registration and ask scripts are included in the envelope's directory listing. Any prompt with one checkable answer serves — that remains the point of shipping the script.