An evaluation of typed decision inference across precision, calibration, latency, and state dependence
LintLabs Research
LintLabs · https://lintlabs.org
Correspondence: [email protected]
Typed decision interfaces can convert an open causal language model into a classifier without retraining: a problem is rendered as a state, a question, and labelled options; one forward pass supplies the next-token probability of each label; and those probabilities are renormalised over the available options. This study evaluated that mechanism in five inference legs on 231 public JevBench items. The strongest measured one-pass leg, Qwen3.8-27B GGUF Q8_0, obtained 82.0 intelligence, 86.1 calibration, and 82/111 (73.9%) hard-tier accuracy. Across the five legs, measured intelligence ranged from 27.4 to 82.6, calibration from 40.2 to 86.9, and hard-tier accuracy from 39.6% to 74.8%.
For the principal recovery experiment, the 29 hard items missed by the Q8_0 one-pass readout were re-rendered with identical prompts and allowed greedy generation with a thinking budget of up to 4000 tokens. The generated answers matched the correct option on 23 of 29 items, implying hard accuracy of 82/111 (73.9%) to 105/111 (94.6%), hard chance-corrected accuracy of 60.6 to 91.9, and intelligence of 82.0 to ≈95.0. The ≈95.0 value is an upper bound, not a measured reasoned score. Only the 29 failures were re-attempted, so the calculation assumes that reasoning never breaks any of the 82 instant answers that were already correct; a reasoning model can instead talk itself out of a correct instant answer. Moreover, 3 of the 29 generations reached the 4000-token cap mid-reasoning, making 23 a floor on the recoverable set. The true reasoned result requires reasoned-mode evaluation of all 111 hard items, which was not performed.
Exact two-sided McNemar tests found the three Qwen3.8-27B precision and inference-stack variants statistically indistinguishable on paired hard-tier outcomes. The study also corrected an earlier informal claim that calibration was the precision-sensitive axis: MLX 4-bit calibration of 86.9 was approximately equal to GGUF Q8_0 at 86.1, whereas GGUF Q4_K_M at 77.2 was the outlier, a pattern consistent with a quantisation-scheme rather than bit-width effect. State-blind controls, family-specific errors, and latency measurements further showed that option-list priors, multi-step computation, and long-context prefill each constrained a different part of the observed system.
Keywords: typed decision models, language-model inference, probability readout, calibration, quantisation, JevBench
A typed decision interface returns a value drawn from a declared option set together with a probability, rather than free-form prose. The implementation studied here, myNameisJevToo [1], rendered each task as a state, question, and list of labelled options. It then made one causal-language-model forward pass, read the probability assigned to each label letter as the next token, and renormalised the selected label probabilities. It generated no intermediate reasoning or answer text. This is an attractive interface because it can be applied to an open model locally, without retraining, fine-tuning, or new weights.
That economy creates a measurement problem. A single next-token distribution may contain enough semantic signal to select familiar or directly stated alternatives, yet it provides no explicit computation trace for arithmetic, probability, temporal rules, or multi-stage policy application. An observed error may therefore reflect the base model, the label interface, the option order, the inference implementation, or an inadequate computational budget. Accuracy alone cannot distinguish these possibilities. Calibration and latency introduce further ambiguity because both the quantisation scheme and the inference stack can affect probabilities and execution time independently of model parameter count.
This paper answered two questions. First, what performance did the one-pass label readout achieve across model scale, quantisation, and inference stack on the available JevBench task set [2]? Second, which constraints belonged to the model and which arose from the readout interface, option construction, or execution regime? The study addressed the second question through reasoned re-attempts of observed hard-tier failures, paired tests among 27B-class legs, a state-blind control, family-level error analysis, binary label-order probes, readout-integrity checks, and latency distributions.
The contributions were:
min_p
distortion and documenting the independent inference-stack failures that
prevented a Xing4.0-29B-A4B leg.Section 2 positions the interface and benchmark. Sections 3 and 4 define the protocol, instrument, systems, scoring, and artefacts. Section 5 reports the findings. Section 6 relates the findings into a mechanism account, followed by limitations, threats to validity, and conclusions.
TypeSafe AI presents Jev through a System One framing: a decision model should return a typed answer rapidly rather than produce an unconstrained textual response [4]. The present work did not evaluate the internals of the vendor’s Jev model. It evaluated an open implementation that imitates the external typed-decision interface by extracting option-label probabilities from ordinary causal language models [1]. This distinction matters: a label readout over an open model is a particular instrument, not evidence that the vendor model uses the same mechanism.
JevBench supplies the frozen decision items, tier organisation, and scoring formulas reproduced in this study [2]. The benchmark distinguishes easy, standard, judge, and hard tiers and defines intelligence, calibration, speed, cost, and a combined score. Only public items were available here, and the judge tier was unavailable. Consequently, this study renormalised the published tier weights across the three observed tiers and excluded cost. Its reported score is therefore a three-axis score equivalent, not a JevBench Score. Published board figures provide context only; they were produced by vendors using their own complete boards and harnesses and are not comparable measurements from this study.
The jev-calibration-audit demonstrated why an option-preserving, state-blind control was necessary [3]. Its published MMLU-ProX result, quoted by the implementation studied here, showed that replacing state with a placeholder while retaining options could still yield 0.383 / 0.463 against chance near 0.15. The present study did not reproduce that dataset or claim. It adopted the underlying audit principle and ran the control on each of its five JevBench legs, thereby measuring how much hard-tier performance remained available from option wording and priors alone.
The inference implementations also formed part of the measured
system. GGUF models ran through llama.cpp [5], including
PrismML’s prebuilt fork for the ternary quantisation. MLX legs used
mlx-lm [7] on Apple’s MLX array framework [6]. The base and
distributed model artefacts were Qwen3.8-27B [8], Ternary-Bonsai-2-27B
[9], and MiniCPM5-2B [10]. Since probability extraction and latency
depend on the stack, this paper reports results by complete leg rather
than treating quantisation as the only varying factor.
Each frozen item was converted into a prompt containing the state,
question, and labelled answer options. Labels were formatted as
single-token strings with a leading space, such as A,
B, and C. The model performed one forward pass
over the rendered prompt. At the answer position, the evaluator obtained
the probability mass assigned to each permitted label letter. If the raw
probabilities for the permitted labels were collected in a vector, each
decision share was computed as raw / raw.sum(). The highest
renormalised share determined the typed answer, while the share itself
supplied its confidence.
This procedure differs from ordinary multiple-choice generation. It does not ask the model to emit a label, explanation, or chain of reasoning. Probability outside the permitted label set is discarded by renormalisation. The measured object is therefore the combination of prompt renderer, label vocabulary, causal model, quantisation, inference stack, and one-step readout rule. The resulting score should not be interpreted as an unconstrained measure of everything the underlying model could answer under another decoding protocol.
The F1 experiment selected the 29 hard-tier items that the Qwen3.8-27B GGUF Q8_0 one-pass readout had answered incorrectly. Their prompts were re-rendered identically: the state, question, labelled options, and their ordering were unchanged. Instead of stopping at the next-token label probabilities, the same model was allowed to generate greedily with a thinking budget of up to 4000 tokens.
The generated response was matched back to the option set. The evaluator recorded the expected option, the option selected by the one-pass readout, the option expressed by the generation, whether that generated selection was correct, the stop type, and the transcript. This matching step converted an open textual trace into the same categorical outcome used by the benchmark. It also introduced a judgement boundary absent from direct label scoring; item-level matching is therefore considered as a threat to validity below.
The protocol was deliberately diagnostic. It held the prompt and model fixed while changing the computation permitted after the prompt. However, it was applied only to observed failures. It can count recovered failures but cannot measure full reasoned-mode accuracy because it contains no observations of whether reasoning preserves or damages the original correct answers.
The label readout was checked at the token and implementation levels.
The label strings A through I were verified to
be single tokens. Label coverage was 100%, with 0 missing labels out of
1698. A cached-prefix implementation was compared with full re-prefill:
it was 4.2x faster and bit-identical, with maximum absolute change in
mean log probability of 0.00000000 nats. These checks support the
integrity of the extracted label scores and rule out token
fragmentation, missing labels, and cache-induced numerical drift as
explanations for the benchmark outcomes.
Two-option tasks nevertheless exposed a structural label-position
trap. Across 74 public binary items, accuracy was 54.1% in forward order
and 52.7% in reversed order, against 50% chance. The similarity is not
evidence of content robustness: a system that always selects the first
position also scores approximately 50% under either ordering. On a
binary item whose state directly contained the answer, the readout chose
the first option in both orders. With state ['no','yes'],
it returned “no” 91.5% of the time; with ['yes','no'], it
returned “yes” 99.2% of the time. When the state was withheld, the
corresponding values were 94.7% and 89.3%. Thus these binary items
carried no detectable content signal for this readout.
The failure was specific to the small option set. Four- and five-option easy items obtained 77.1% against 28.4% chance. Accordingly, the benchmark signal analysed here came from its four- and five-option items, whereas the binary probe was treated as an instrument diagnostic rather than corroborating task evidence.
All 27B-class legs ran on a Mac Studio M3 Ultra with 96 GB of memory.
The 2B baseline ran on a Mac mini M4 with 16 GB. The five measured legs
were Qwen3.8-27B GGUF Q8_0, Qwen3.8-27B GGUF Q4_K_M, Qwen3.8-27B MLX
4-bit, Ternary-Bonsai-2-27B PQ2_0, and MiniCPM5-2B MLX 4-bit. The
explicitly recorded weight sizes were 27 GB for Qwen Q8_0 and 6.7 GB for
Ternary-Bonsai PQ2_0. Model cards identify the 27B and 2B parameter
classes [8], [9], [10]. GGUF inference used llama.cpp; MLX
inference used mlx-lm. Ternary-Bonsai PQ2_0 loaded only
through PrismML’s prebuilt llama.cpp fork.
An intended Xing4.0-29B-A4B leg was tested and failed at the loading
stage, rather than remaining an open evaluation. Two independent stack
checks agreed. llama.cpp reported
unknown model architecture: 'xing4_0', and inspection of
its src/llama-arch.cpp on master found neither
a xing nor telechat architecture entry.
Separately, mlx-lm shipped telechat3, the
predecessor architecture, but its model registry contained no
xing4_0 module. Because the model could not be run by
either stack, it contributed no measurements.
The task set comprised all 231 public JevBench items: 48 easy, 72 standard, and 111 hard. These were only the public halves: the benchmark contains 48/72 public easy items, 72/96 public standard items, and 111/220 public hard items. The judge tier and private halves were unavailable. The evaluation therefore describes performance on the public subset, not on the complete benchmark.
Per-tier raw accuracy was converted to chance-corrected accuracy as
(accuracy - chance) / (1 - chance). The benchmark chance
rates were 0.284 for easy, 0.317 for standard, and 0.336 for hard.
Published tier weights are 0.14 for easy, 0.28 for standard, 0.28 for
judge, and 0.30 for hard. Since judge items were unavailable,
intelligence was the weighted mean of the three observed
chance-corrected tier scores after renormalising the easy, standard, and
hard weights.
Calibration was derived from hard-tier expected calibration error.
Speed was derived from latency under JevBench’s own-server rule,
adjusted = raw x 2 + 0.15 s. This adjustment is a benchmark
assumption, not an independently measured delay. The reported combined
value used intelligence, calibration, and speed. The cost axis was
excluded, so the resulting three-axis score cannot be called a JevBench
Score.
Differences among the three Qwen3.8-27B legs were tested on paired hard-tier correctness using the exact two-sided McNemar test [11]. This design used item pairing rather than treating the outcomes as independent samples. It tested equality of the marginal error rates observable in 111 paired hard items; it did not estimate a general performance hierarchy beyond this public set.
Every reported benchmark number derived from archived JSON artefacts.
Per-item and summary files under artefacts/results/ cover
the Q8_0, Q4_K_M, MLX 4-bit, Ternary-Bonsai, MiniCPM5, binary-order,
token-scoring, quantisation, and state-blind runs. The
artefacts/reasoned/ directory contains 29 raw generation
transcripts and reasoned_summary.json, which records
correctness and stop type for every reasoned re-attempt.
A suspected min_p artefact was also tested and
disproved. The concern was that llama.cpp might apply
min_p = 0.05 by default and truncate low-probability
labels. Top-logprob outputs were identical with no min_p,
with min_p = 0.05, and with min_p = 0.0, on
both stock and forked builds. Thus this setting was not the cause of the
readout results. The raw probe output was not archived as a committed
artefact, however, so this is a design note rather than a reproducible
measurement.
Table 1 reports all five measured legs on the same 231 public items. The three Qwen3.8-27B legs clustered near the top, but their order depended on the axis: Q4_K_M had the highest intelligence, MLX 4-bit the highest calibration and speed, and Q8_0 the highest three-axis score. Ternary-Bonsai retained perfect easy-tier accuracy but declined more sharply on standard and hard items. The MiniCPM5-2B baseline separated clearly from the 27B-class legs.
Table 1. Measured results on the 231 public JevBench items. Score is the three-axis equivalent over intelligence, calibration, and speed; it excludes cost and is not a JevBench Score.
| Leg | Intelligence | Calibration | Speed | Score | Easy | Standard | Hard |
|---|---|---|---|---|---|---|---|
| Qwen3.8-27B GGUF Q8_0 | 82.0 | 86.1 | 66.4 | 77.7 | 100.0% | 97.2% | 73.9% |
| Qwen3.8-27B GGUF Q4_K_M | 82.6 | 77.2 | 65.9 | 74.9 | 100.0% | 97.2% | 74.8% |
| Qwen3.8-27B MLX 4-bit | 77.6 | 86.9 | 67.4 | 76.9 | 100.0% | 94.4% | 69.4% |
| Ternary-Bonsai-2-27B PQ2_0 (6.7 GB) | 67.2 | 81.2 | 66.3 | 71.2 | 100.0% | 86.1% | 60.4% |
| MiniCPM5-2B MLX 4-bit | 27.4 | 40.2 | 73.6 | ≈13 | 77.1% | 50.0% | 39.6% |
The corresponding hard-tier counts were 82/111 for Q8_0, 83/111 for Q4_K_M, 77/111 for MLX 4-bit, 67/111 for Ternary-Bonsai, and 44/111 for MiniCPM5-2B. Hard-tier ECE was respectively 0.0693, 0.1139, 0.0655, 0.0940, and 0.2989.
For scale only, Table 2 records published vendor reference figures reproduced by the JevBench board [2]. They are not measurements from this study and are not comparable to Table 1: each vendor used its own harness and full board. They are separated from the measured table to prevent a shared-column comparison.
Table 2. Vendor-published JevBench reference figures from vendors’ own harnesses and full boards; not measurements from this study and not comparable to Table 1.
| Vendor system | Intelligence | Calibration | Speed | Score |
|---|---|---|---|---|
| Jev 1.13.0 (TypeSafe AI) | 85.7 | 82.7 | 83.3 | 74.4 |
| SemIf (Qwen3.5-4B) | 79.0 | 72.6 | 83.7 | 73.1 |
| jeff (GLiFormer 400M) | 46.9 | 64.6 | 63.5 | 54.4 |
The Q8_0 readout answered 82 of 111 hard items correctly and 29 incorrectly. When the identical prompts for those 29 failures were re-rendered and the model generated greedily with up to 4000 thinking tokens, 23 of the 29 generated answers matched the correct option. Applying those recovered outcomes to the original count implies a transition from 82/111 (73.9%) to 105/111 (94.6%) hard accuracy, from 60.6 to 91.9 hard chance-corrected accuracy, and from 82.0 to ≈95.0 intelligence.
The ≈95.0 intelligence value is an upper bound, not a measured reasoned score. Only the 29 failures were re-attempted. The implied result therefore assumes that reasoning never changes any of the 82 correct one-pass answers into an error, even though a reasoning model can talk itself out of a correct instant answer. In addition, 3 of the 29 attempts hit the 4000-token cap mid-reasoning, so 23 is a floor on the recoverable set. A true reasoned-mode figure requires running all 111 hard items in reasoned mode; that run was not performed.
Exact two-sided McNemar tests did not resolve a difference among the three Qwen3.8-27B legs. Q8_0 versus Q4_K_M yielded p = 1.000, with 82 versus 83 correct and discordant-pair counts of 2 versus 3. Q8_0 versus MLX 4-bit yielded p = 0.267, with 82 versus 77 correct and discordant counts of 9 versus 4. Q4_K_M versus MLX 4-bit yielded p = 0.238, with 83 versus 77 correct and discordant counts of 12 versus 6.
There was therefore no measured precision ladder on intelligence. In the 111-item hard tier, one item corresponds to 0.9% raw accuracy and 0.57 intelligence points. The Q8_0 and Q4_K_M legs differed by a single item. At this resolution, the benchmark could not distinguish these 27B-class variants.
The calibration results corrected an earlier informal claim that “calibration is the precision-sensitive axis.” That claim had been over-generalised from a single pair and was wrong. MLX 4-bit reached 86.9 calibration, approximately equal to GGUF Q8_0 at 86.1, while GGUF Q4_K_M at 77.2 was the outlier. The observed pattern is therefore consistent with a quantisation-scheme and inference-stack effect rather than a general bit-width effect. Hard-tier ECE supports the same ordering: 0.0655 for MLX 4-bit, 0.0693 for Q8_0, and 0.1139 for Q4_K_M.
Table 3 partitions Q8_0 hard-tier correctness by benchmark family.
The two lowest-performing families were temporal_numeric
and probability. Both require multi-step computation over
values in the state. By contrast, multi_hop occupied the
middle of the table, preventing the pattern from being reduced to a
generic inability to reason.
Table 3. Qwen3.8-27B GGUF Q8_0 hard-tier accuracy by family.
| Family | Correct / n | Accuracy |
|---|---|---|
| temporal_numeric | 6/15 | 40.0% |
| probability | 6/10 | 60.0% |
| tradeoff | 4/6 | 66.7% |
| long_policy | 14/19 | 73.7% |
| judge_hard | 13/17 | 76.5% |
| multi_hop | 14/18 | 77.8% |
| trap | 7/8 | 87.5% |
| ambiguous | 7/7 | 100.0% |
| adversarial | 6/6 | 100.0% |
| routing_hard | 5/5 | 100.0% |
Table 4 shows both measured and benchmark-adjusted latency. Ternary-Bonsai PQ2_0 occupied 6.7 GB, compared with 27 GB for Qwen Q8_0, yet their raw p50 values were 600 ms and 589 ms and their raw p95 values were 8684 ms and 8519 ms. Expressed at the precision used for the cross-size comparison, the p95 values were 8.7 s and 8.5 s. The long tail instead tracked long-context prefill: hard-tier states ran about 3,700 tokens.
Table 4. Measured raw latency and latency adjusted under JevBench’s own-server assumption.
| Leg | Raw p50 | Raw p95 | Adjusted p50 | Adjusted p95 |
|---|---|---|---|---|
| Qwen3.8-27B GGUF Q8_0 | 589 ms | 8519 ms | 1.33 s | 17.19 s |
| Qwen3.8-27B GGUF Q4_K_M | 637 ms | 9019 ms | 1.42 s | 18.19 s |
| Qwen3.8-27B MLX 4-bit | 465 ms | 8437 ms | 1.08 s | 17.02 s |
| Ternary-Bonsai-2-27B PQ2_0 | 600 ms | 8684 ms | 1.35 s | 17.52 s |
| MiniCPM5-2B MLX 4-bit | 188 ms | 4092 ms | 0.53 s | 8.33 s |
The adjusted values in Table 4 are not additional timing
observations. They are transformations of the raw values under the
benchmark rule raw x 2 + 0.15 s.
The state-blind control replaced state with a placeholder while leaving each question and option list unchanged. Table 5 reports both raw hard accuracy in that condition and the proportion of the original state-present hard accuracy retained. The four 27B legs retained 70.7% to 75.3% of their hard accuracy. MiniCPM5-2B performed better without state, at 45.9% rather than 39.6%, or 116% retained. This inversion indicates reliance on option-list priors rather than successful state reading.
Table 5. State-blind accuracy by tier and hard-tier accuracy retained relative to the state-present run.
| Leg | Easy | Standard | Hard | Hard accuracy retained |
|---|---|---|---|---|
| Qwen3.8-27B GGUF Q8_0 | 29.2% | 30.6% | 52.3% | 70.7% |
| Qwen3.8-27B GGUF Q4_K_M | 33.3% | 30.6% | 53.2% | 71.1% |
| Qwen3.8-27B MLX 4-bit | 35.4% | 30.6% | 52.3% | 75.3% |
| Ternary-Bonsai-2-27B PQ2_0 | 33.3% | 33.3% | 44.1% | 73.1% |
| MiniCPM5-2B MLX 4-bit | 31.3% | 27.8% | 45.9% | 116% |
The family distribution and recovery experiment identify a specific
constraint. temporal_numeric accuracy was 40.0%, and
probability accuracy was 60.0%, the two lowest Q8_0 family
results. These tasks require values to be extracted from state,
transformed through arithmetic or probability rules, and compared with
labelled outcomes. A one-pass next-label readout must express the result
at the answer position without being permitted to externalise those
intermediate transformations. Greedy reasoned generation changed
precisely that computational allowance while preserving the prompt. Its
recovery of 23 observed misses is therefore consistent with
interface-limited computation rather than absence of the requisite
capabilities from the model.
This account is narrower than saying that one-pass inference cannot
reason. multi_hop achieved 77.8%, above four other
families, and three families achieved 100.0%. The discriminating
property was not the benchmark’s family name or an undifferentiated
concept of difficulty. It was the need to carry out multi-step
arithmetic or probabilistic computation over state values before mapping
the result to an option. Direct routing, adversarial recognition, and
ambiguity handling could often be expressed in the immediate label
distribution; extended calculation was much less reliable there.
The F1 intervention nevertheless isolates computation budget imperfectly. It compares selected failures under one-pass scoring with generation under a large budget, and generated responses require categorical matching. It establishes that most observed failures were recoverable under that alternate interface. It does not establish the net gain over all hard items or show that every recovery arose from faithful state-based reasoning. Those boundaries motivate the upper-bound treatment rather than weaken the diagnostic observation.
The state-blind results prevent a simple attribution of all correct answers to state comprehension. Q8_0 retained 70.7% of its state-present hard accuracy after state removal, and the other 27B legs retained 71.1%, 75.3%, and 73.1%. The option list therefore exposed substantial answerable structure: linguistic plausibility, answer-type constraints, or dataset regularities could support a choice without the state. MiniCPM5-2B made the issue more explicit by improving from 39.6% with state to 45.9% without it.
At the same time, the state-present and state-blind levels were not identical for the 27B legs. Q8_0 fell from 73.9% to 52.3%, Q4_K_M from 74.8% to 53.2%, MLX 4-bit from 69.4% to 52.3%, and Ternary-Bonsai from 60.4% to 44.1%. The measured system thus combined state-conditioned signal and option-only signal. The family-level failures plausibly occur where option priors are insufficient and the state must be converted into a computed answer.
The size comparison separates memory compression from end-to-end response time. Reducing the stored weights from 27 GB for Q8_0 to 6.7 GB for Ternary-Bonsai did not reduce p50 or p95 latency: the respective pairs were 589 ms versus 600 ms and 8519 ms versus 8684 ms. Hard states of about 3,700 tokens made prompt processing, rather than the final label readout or weight size alone, the dominant source of tail latency.
This mechanism also explains why a typed interface can be generation-free without being uniformly fast. Eliminating answer generation removes decoding time, but the full state still has to be prefetched. The 4.2x cached-prefix result demonstrates that implementation reuse can accelerate repeated scoring without changing probabilities, yet it cannot eliminate the initial cost of ingesting a long, item-specific state.
F2 constrains claims about quantisation more strongly than the point estimates do. The Q8_0 and Q4_K_M hard results differed by only one item, while a hard item represented 0.9% raw accuracy and 0.57 intelligence points. Their exact paired test produced p = 1.000. The other comparisons also failed to distinguish the variants, at p = 0.267 and p = 0.238. A ranking of 82.6, 82.0, and 77.6 intelligence would therefore overstate what this public set resolves.
Calibration did not follow that apparent intelligence order. The near agreement of MLX 4-bit at 86.9 and Q8_0 at 86.1, alongside Q4_K_M at 77.2, rejects the earlier descriptive shortcut that calibration was generally precision-sensitive. Since both quantisation representation and inference stack changed between GGUF and MLX, the defensible interpretation is a quantisation-scheme effect in the measured systems, not a universal causal claim about a particular library.
The binary probe gives a limiting example of why the readout should be treated as an instrument. Selecting the first option in both orderings produced superficially stable near-chance aggregate accuracy while ignoring a plainly informative state. Reversal therefore did not reveal semantic invariance; it revealed position invariance. Four- and five-option items escaped this degenerate pattern and carried substantial above-chance signal, but the binary result shows that label probability can measure prompt-position preference rather than task knowledge.
Together with the state-blind control, this finding distinguishes two interface effects. Binary position bias fixes attention on a label location, whereas option leakage exploits information in the answer strings. Both can produce correct benchmark responses without using the intended state evidence. Readout-integrity checks show that these effects were not caused by missing or multi-token labels: they are behavioural properties of the interface-model combination.
The F1 calculation requires the strongest qualification in this study. The one-pass readout missed 29 hard items; identical prompts with greedy generation and up to 4000 thinking tokens answered 23 of those 29 correctly. This implies 82/111 (73.9%) to 105/111 (94.6%) hard accuracy, 60.6 to 91.9 hard chance-corrected accuracy, and 82.0 to ≈95.0 intelligence. The ≈95.0 value is an upper bound, not a measured reasoned score. Only failures were re-attempted, so the calculation assumes that reasoning never breaks any of the 82 correct instant answers. A reasoning model can talk itself out of a correct instant answer. Moreover, 3 of the 29 selected items reached the 4000-token cap mid-reasoning, making 23 a floor rather than a point estimate of the recoverable set. The true reasoned figure requires running all 111 hard items in reasoned mode, which was not done.
First, the selected-subset design prevents a direct estimate of reasoned-mode accuracy. None of the originally correct hard items was re-attempted, and therefore neither the frequency nor family distribution of reasoning-induced regressions is known. The upper bound and recovery floor point in opposite directions: regressions could lower the implied total, while completion beyond the cap could recover more of the selected failures. They cannot be combined into a point estimate without the missing full-tier run.
Second, evaluation was restricted to public halves. The study used 48 easy, 72 standard, and 111 hard items; private easy, standard, and hard items and the complete judge tier were unavailable. Public items can differ from withheld items, and model or interface choices can be selected against public outcomes. Intelligence consequently used weights renormalised over only three tiers.
Third, the reported combined score omitted cost. Even where its numerical form resembles the public board’s combined presentation, it is a three-axis equivalent over intelligence, calibration, and speed, not a JevBench Score. Vendor reference figures were produced on vendors’ own full boards with their own harnesses and cannot validate or rank the locally measured legs.
Fourth, the min_p probe is a design note. It produced
identical top log probabilities across no setting, 0.05, and 0.0 on
stock and forked builds, but its raw output was not archived. It
supports the debugging conclusion that min_p was not the
cause, yet it lacks the reproducibility of the committed benchmark and
reasoned artefacts.
Fifth, measurements were single-seed, single-run results without repeated trials. Greedy reasoned decoding removed sampling variation from that arm, but run-to-run system timing variation and any nondeterminism elsewhere were not estimated. Exact paired tests address item-level discordance on the observed hard set; they do not substitute for replication.
Sixth, calibration was task-local. Hard-tier ECE and the derived calibration axis describe confidence alignment on these public JevBench hard items under this option construction. They do not establish calibration on other domains, answer-set sizes, prompt renderers, or deployment distributions. The correction from a bit-width interpretation to a scheme-consistent pattern is likewise observational rather than a controlled decomposition of library, format, and quantiser effects.
Finally, the instrument’s valid scope was narrower than the nominal task collection. Binary items exhibited position selection without content signal and were useful as a diagnostic, not as evidence of typed decision quality. The four- and five-option results carried signal, but state-blind accuracy showed that even those scores combined state use with option-list priors.
The reasoned-mode comparison was not a randomised paired trial over all hard items. Selection was determined by the Q8_0 readout’s errors, guaranteeing room for improvement and preventing observation of regressions among initially correct items. Although identical prompt rendering reduced one source of confounding, changing from next-token scoring to extended generation also changed the output interface and introduced answer matching. A generated discussion could contain several candidate labels, an indirect answer, or an unfinished computation. The archived transcripts make those judgements auditable, but item-level matching remains a potential source of classification error.
Stack and quantisation effects were not factorially isolated. GGUF
Q8_0, GGUF Q4_K_M, and MLX 4-bit differed in representation, and the MLX
comparison also changed the inference implementation. Consequently, F3
supports a measured scheme pattern but does not identify a single causal
software or numerical component. The unarchived min_p probe
also provides weaker internal evidence than the committed runs, despite
having been checked on both stock and forked builds.
Only one public benchmark and three model families were evaluated on two Apple systems. Performance may not generalise to private items, other decision domains, other model sizes, other prompt templates, accelerator types, or serving engines. Benchmark contamination is possible because the public half is available for model training, prompt tuning, or system selection. Even without direct training contamination, iterative development against visible items can inflate apparent generality.
Latency is especially hardware- and stack-specific. The Mac Studio and Mac mini results describe their respective local configurations, and the benchmark’s own-server transformation is an assumption. The observation that long-context prefill dominated the tested tail may transfer conceptually, but the precise latencies and the balance between memory bandwidth, prompt length, and kernels need not transfer.
The label readout is an interface artefact as well as a model probe. It measures the next-token distribution over chosen label tokens after a particular renderer, then conditions that distribution by renormalising over options. It does not directly measure the probability the model would assign to answer contents, nor its best achievable accuracy under generation, tool use, or other deliberation. Binary position bias demonstrates that the construct can collapse into positional preference.
Intelligence, calibration, and speed are benchmark-defined composites rather than exhaustive properties. Renormalising weights after removing the judge tier changes the construct relative to the complete board. Hard-tier ECE measures calibration only within the hard task distribution. The three-axis score omits economic cost. State-blind retained accuracy measures exploitable option information, but it cannot by itself distinguish memorisation, linguistic plausibility, dataset artefacts, or legitimate world knowledge encoded in an option.
McNemar tests appropriately used paired hard outcomes, but 111 items gave coarse resolution. A single item moved raw hard accuracy by 0.9% and intelligence by 0.57 points, so small variant differences were intrinsically difficult to detect. The reported non-significant p-values establish failure to resolve a difference; they do not prove practical equivalence or identical performance in a larger population.
Multiple pairwise tests were descriptive and no repeated benchmark samples existed. Family partitions were smaller still, so their accuracies identify concentrations in this task set rather than precise population rates. The recovery count of 23/29 came from a selected failure set and must not be assigned the inferential meaning of an unbiased success probability over all hard items.
On 231 public JevBench items, one-pass option-label readout provided a cheap local typed decision interface without retraining, but its strongest measured leg reached 73.9% on the hard tier despite perfect easy-tier performance. Error concentration in temporal-numeric and probability tasks, followed by recovery of 23 of 29 observed Q8_0 misses under an identical-prompt thinking budget, established that the one-pass interface was a binding constraint for many computational items. The implied ≈95.0 intelligence remains an upper bound, not a measured reasoned result.
The study further established that the public hard tier could not
statistically resolve the tested Qwen3.8-27B variants, that much hard
accuracy survived removal of state, and that the latency tail reflected
long-context prefill rather than model weight size alone. It corrected
the earlier, wrong generalisation that calibration was inherently
precision-sensitive: the measured MLX 4-bit and GGUF Q8_0 values were
similar, with GGUF Q4_K_M as the outlier, consistent with a
quantisation-scheme effect. The suspected min_p distortion
was tested and disproved, while Xing4.0-29B-A4B failed independent
loading checks in both inference stacks.
The most important remaining measurement is direct and singular: run all 111 hard items in reasoned mode. Only that complete paired evaluation can measure both recoveries and reasoning-induced regressions and replace the current upper bound with a reasoned-mode score.