JevBench · measurement note
On 231 public JevBench items, a single forward pass that never generates a token was the binding constraint on hard-tier questions — not model quality. Re-running the same prompts on the questions it got wrong, with a thinking budget, recovered most of them.
The harness reads the model's probability mass on each option's letter in one forward pass. No token generation at all — the output is a typed decision with a probability. That design is what makes it cheap, and it is also what caps it.
29 hard items were answered wrongly by the one-pass readout. Re-running the same rendered prompts with a thinking budget (greedy, up to 4000 tokens) got 23 of those 29 correct.
| Leg | Intel | Calib | Speed | Score | Easy | Standard | Hard |
|---|---|---|---|---|---|---|---|
| Qwen3.8-27B GGUF Q8_0 | 82.0 | 86.1 | 66.4 | 77.7 | 100.0% | 97.2% | 73.9% |
| Qwen3.8-27B GGUF Q4_K_M | 82.6 | 77.2 | 65.9 | 74.9 | 100.0% | 97.2% | 74.8% |
| Qwen3.8-27B MLX 4-bit | 77.6 | 86.9 | 67.4 | 76.9 | 100.0% | 94.4% | 69.4% |
| Ternary-Bonsai-2-27B PQ2_0 (6.7 GB) | 67.2 | 81.2 | 66.3 | 71.2 | 100.0% | 86.1% | 60.4% |
| MiniCPM5-2B MLX 4-bit | 27.4 | 40.2 | 73.6 | ~13 | 77.1% | 50.0% | 39.6% |
Do not read a spread into the 27B rows. McNemar on paired hard-tier results: Q8_0 vs Q4_K_M p = 1.000; Q8_0 vs MLX 4-bit p = 0.267; Q4_K_M vs MLX 4-bit p = 0.238. With n = 111 on hard, one item is 0.9% raw ≈ 0.57 intelligence points — Q8_0 (82/111) vs Q4_K_M (83/111) is literally a one-item difference.
temporal_numeric and
probability — are exactly the ones that need multi-step computation, which a single forward pass
cannot perform.Calibration tracks the quant scheme, not bit-width. MLX 4-bit scores 86.9 — about equal to GGUF Q8_0 at 86.1. The GGUF Q4_K_M leg (77.2) is the outlier. An earlier claim that calibration is "the precision-sensitive axis" was over-generalised from a single pair and is wrong.
Latency is size-independent. Bonsai is 6.7 GB, Qwen Q8_0 is 27 GB — p50 600 vs 589 ms, p95 8.7 s vs 8.5 s. The tail is long-context prefill (hard-tier states run ~3,700 tokens), not weight size. Q8_0 raw p95 8519 ms; adjusted per JevBench's own-server rule (raw ×2 + 0.15 s) p50 1.33 s / p95 17.19 s.
With the state replaced by a placeholder and the options kept: the 27B legs hold on to ~71–73% of their hard-tier accuracy, while the 2B scored better without the state — 45.9% vs 39.6% — which looks like an option-list prior rather than state grounding.
unknown model architecture: 'xing4_0'. Ternary-Bonsai's PQ2_0 does load, on PrismML's prebuilt fork.Public JevBench task items only — easy 48, standard 72, hard 111. Scoring uses JevBench's own formulas:
per-tier accuracy, chance-corrected (acc − chance) / (1 − chance), tier weights
easy 0.14 / standard 0.28 / judge 0.28 / hard 0.30. The intelligence axis is the weighted mean of the
chance-corrected tier scores. Other axes: calibration (hard-tier ECE) and speed (latency, using JevBench's
own-server rule of raw ×2 + 0.15 s). All 27B legs ran on a Mac Studio M3 Ultra 96 GB; the 2B baseline on a
Mac mini M4 16 GB.