Benchmarks
More
OverviewBenchmaxxingCompareChartsCost vs CapabilityProviders per ModelProvider explorerGatewaysEU & SovereignJev-class modelsImage JevBenchAbout
Official JevBench release v1.5.1

JevBench v1.5.1 — Jev alternatives ranking

JevBench is Benchmark Heaven's own benchmark for Jev-class decision models: state and a bounded rubric in, a typed answer out.

Frozen release · 1,624 decisions per system (904 open + 720 sealed; sealed decisions are half of Intelligence) · 98 ranked of 103 roster systems · only system-level sealed aggregates are published · aggregate results JSON sha256 6f2fa547454b…

Making decisions from images? Explore Image JevBench v0.1.4 and compare its systems.

Share this version · View live board · Previous release: JevBench v1.5.0

JevBench v1.5.1 · headline ranking

Capability ranking of Jev-class systems

Capability averages Intelligence and Calibration. Jev 1.13.0 leads the Jev-class systems with 80.0.

Jev-class means at most 2× Jev's cost and median latency. How we choose ↘

#SystemICCap.$/1k
  1. 1Jev 1.13.072.054.780.0$0.032*Reference system for the Jev-class limits
  2. 2Winnow-12B Q874.456.679.3$0.028*
  3. 3Cygnet71.156.479.0$0.028*
  4. 4Surogate Rune 26B-A4B v369.749.079.0$0.050*
  5. 5Jev-Omni70.556.176.5$0.029*
  6. 6djev72.348.276.4$0.053*
  7. 7JevK5 v0.356.363.172.3$0.017*
  8. 8Plumb-4B55.863.171.6$0.017*
  9. 9Decision 4B v1.253.763.171.1$0.017*
  10. 10Imajev-4B53.563.370.8$0.017*
Show all 55 Jev-class systems (45 more)
  1. 11decider-4b v255.864.570.7$0.015*
  2. 12Decision 4B v1.153.163.170.2$0.017*
  3. 13Hopper49.962.368.9$0.018*
  4. 14Malkuth-4B54.555.868.8$0.030*
  5. 15metask-jev-4b53.557.768.1$0.026*
  6. 16jqv49.151.267.9$0.042*
  7. 17SemIf51.363.167.6$0.017*
  8. 18jev-local56.358.967.0$0.024*
  9. 19spark-s1-4b-v662.160.465.9$0.021*
  10. 20JevK5 v0.2.046.763.165.8$0.017*
  11. 21Decision 2B31.366.458.8$0.013*
  12. 22local-jev Qwen3.5-4B33.759.358.1$0.023*
  13. 23open-alternative-jev37.463.357.1$0.017*
  14. 24decider-2b42.364.956.9$0.015*
  15. 25system-one-open41.668.356.6$0.011*
  16. 26JEV Qwen3.5-9B Base NVFP432.247.656.6$0.056*
  17. 27Malkuth-2B35.565.655.4$0.014*
  18. 28typecastlm31.264.054.0$0.016*
  19. 29kev 4B39.965.853.8$0.014*
  20. 30Raw Qwen3 4B Instruct 2507 direct logits54.163.453.5$0.017*
  21. 31Raw Phi-4 mini direct logits27.653.249.5$0.036*
  22. 32Qwen3-Reranker-4B21.048.448.6$0.052*
  23. 33decision-machine-114.856.348.0$0.029*
  24. 34Certo v10.096.943.8$0.0013*
  25. 35Mixedbread mxbai-rerank-base-v20.060.343.3$0.021*
  26. 36ZeroEntropy zerank-24.848.443.2$0.052*
  27. 37Von0.082.741.7$0.0038*
  28. 38BAAI bge-reranker-v2-m30.059.341.7$0.023*
  29. 39openJev Verdict 1.40.086.640.2$0.0028*
  30. 40kev 0.6B10.380.139.0$0.0046*
  31. 41lev-350m0.080.038.8$0.0046*
  32. 42Open Jev JSON Canvas77.149.338.6$0.049*
  33. 43smalljev semantic-v93.760.838.4$0.020*
  34. 44Decision Fast0.080.138.0$0.0046*
  35. 45Alibaba GTE Reranker ModernBERT-base0.069.337.6$0.011*
  36. 46OpenDecision2.179.137.3$0.0050*
  37. 47JevAct11.268.636.9$0.011*
  38. 48GLiNER2.5 small9.286.632.5$0.0028*
  39. 49kev 0.5B0.080.132.5$0.0046*
  40. 50CLM-8B11.450.530.0$0.045*
  41. 51verdict-small0.0100.029.7$0.00087*
  42. 52openJev Verdict0.086.626.1$0.0028*
  43. 53GLiNER213.586.624.5$0.0028*
  44. 54Raw Qwen3 0.6B direct logits10.677.616.0$0.0056*
  45. 55Raw Qwen3 1.7B direct logits9.768.515.6$0.011*

Wide coloured bar = Capability (0–100). Thin red line = cost per 1,000 decisions; log scale, each gridline = 10×, shorter is cheaper. * = est. (estimated cost). # counts ranked Jev-class systems; “–” marks unranked or outside systems. Tap ⓘ for the full values, median latency and cost relative to Jev.

  • Jev (TypeSafe, closed)
  • Jev rebuild
  • Instruction model, JSON schema
  • Small tool-calling model
  • Zero-shot classifier
  • Closed decision API
  • Reranker (neutral adapter)
  • Raw-logit control (base model)
  • Native-logit decision engine
  • Unclassified
  • system-one-open
  • Cost line
Show general-purpose LLMs and other systems outside the limits

Sorted by Capability, not numbered. Each row says which limit it misses, measured against Jev 1.13.0 ($0.032 per 1,000 decisions, median 0.62 s).

  1. –Qwen3.8 27B95.60.096.8$2.18*Outside: cost 67.4× Jev, latency 10.5× Jev
  2. –GPT-6 Luna (medium)96.238.395.9$0.11*Outside: cost 3.5× Jev, latency 2.5× Jev
  3. –DeepSeek V4.1 Flash93.719.195.3$0.50*Outside: cost 15.4× Jev, latency 2.9× Jev
  4. –GPT-6 Luna (low)95.339.195.1$0.11*Outside: cost 3.3× Jev, latency 2.6× Jev
  5. –GPT-5.6 Luna94.330.794.5$0.20Outside: cost 6.3× Jev, latency 2.1× Jev
  6. –djev77.328.186.5$0.25*Outside: cost 7.7× Jev, latency 2.4× Jev
  7. –OpenJev84.228.583.7$0.24*Outside: cost 7.5× Jev, latency 2.5× Jev
  8. –Autoloops – Gemma 4 31B IT76.739.681.3$0.10Outside: cost 3.2× Jev
  9. –Eikos-27B75.128.780.7$0.24*Outside: cost 7.4× Jev
  10. –AutoJev-27B72.829.480.2$0.23*Outside: cost 7.0× Jev
  11. –SimpleJev Qwen3.8-27B72.814.980.0$0.69*Outside: cost 21.3× Jev, latency 2.7× Jev
  12. –AutoJev-27B72.829.479.7$0.23*Outside: cost 7.0× Jev
  13. –swanOne71.242.279.1$0.085*Outside: cost 2.6× Jev
  14. –NInfer Qwen3.8-Flash-Next mixed67.242.577.9$0.082*Outside: cost 2.5× Jev
  15. –Gemini 3.1 Flash-Lite77.629.876.1$0.22Outside: cost 6.8× Jev
  16. –NInfer Qwen3.8-27B NVFP465.529.175.7$0.23*Outside: cost 7.1× Jev
  17. –reflex-27b62.825.874.4$0.30*Outside: cost 9.2× Jev, latency 4.5× Jev
  18. –Instinct62.729.273.9$0.23*Outside: cost 7.1× Jev
  19. –NInfer Qwen3.8-27B NVFP461.129.173.8$0.23*Outside: cost 7.1× Jev
  20. –Open-Jev 9B63.833.172.7$0.17*Outside: cost 5.3× Jev, latency 2.1× Jev
  21. –LitJev58.328.471.4$0.24*Outside: cost 7.6× Jev, latency 5.0× Jev
  22. –Standard One 8B59.643.371.4$0.078*Outside: cost 2.4× Jev
  23. –decider-35b-a3b60.534.471.2$0.15*Outside: cost 4.8× Jev
  24. –openjev-sglang58.635.670.8$0.14*Outside: cost 4.3× Jev
  25. –Qwen3.5-9B Jev-like data-mix v260.445.770.7$0.065*Outside: cost 2.0× Jev
  26. –Bespoke Nimble 9B63.736.870.5$0.13*Outside: cost 4.0× Jev
  27. –JevOne54.139.869.5$0.10*Outside: cost 3.1× Jev
  28. –reflex 4B51.563.169.2$0.017*Outside: latency 4.6× Jev
  29. –kev 8B48.340.459.7$0.097*Outside: cost 3.0× Jev
  30. –Open-Jev 2B33.633.153.6$0.17*Outside: cost 5.3× Jev
  31. –OpenSourceJev25.869.150.9$0.011*Outside: latency 2.3× Jev
  32. –Raw Qwen3 8B direct logits51.145.550.2$0.065*Outside: cost 2.0× Jev
  33. –system-one50.545.049.9$0.068*Outside: cost 2.1× Jev
  34. –jeff4.481.142.4$0.0043*Outside: latency 11.6× Jev
  35. –open-jev-deberta-v3-large0.077.638.5$0.0056*Outside: latency 4.8× Jev
  36. –Laya0.084.936.9$0.0032*Outside: latency 2.4× Jev
  37. –Qwen3.5-0.8B Decision Model0.079.536.8$0.0048*Outside: latency 2.1× Jev
  38. –GLiNER2.5 multi14.086.636.0$0.0028*Outside: latency 2.2× Jev
  39. –GLiNER2 large22.577.632.4$0.0056*Outside: latency 3.0× Jev
  40. –SimpleJev16.770.331.7$0.0098*Outside: latency 12.0× Jev
  41. –Mirror5.689.324.4$0.0023*Outside: latency 7.4× Jev
  42. –Needle 30.061.60.0$0.019*Outside: latency 219.8× Jev
  43. –Needle 3, options as tools0.061.60.0$0.019*Outside: latency 94.3× Jev

Jev-class = cost per decision at most 2× Jev 1.13.0's (≤ $0.065 per 1,000 decisions) and median latency at most 2× Jev 1.13.0's (≤ 1.23 s, the adjusted p50 — the same median the speed chart plots, not the four-axis Speed score). 55 of 98 systems qualify; the other 43, including the general-purpose LLMs, are listed below the divider in the ranking. The charts below show speed and cost beside Capability; the official JevBench Score weighs all four axes.

Capability against cost and speed

Jev-class systems are shown by default. Bubble size follows the official JevBench Score. The five most capable Jev-class systems are labelled.

Capability vs cost

Upper right is best: more capable and cheaper. Cost is USD per 1,000 decisions on a log scale. The dashed line is 2× Jev’s cost.

Capability vs cost: 55 systems. Upper right is best: more capable and cheaper.102030405060708090100$0.0010$0.010$0.10$ per 1,000 decisions (log)Capability↑2× Jev← priciercheaper →1. Jev 1.13.02. Winnow-12B Q83. Cygnet4. Surogate Rune 26B-A4B v35. Jev-Omni
55 systems. Tap a bubble for its values.

Capability vs speed

Upper right is best: more capable and faster. Speed here is the median-latency speed — the same adjusted median (p50) latency the Jev-class limit uses — a log scale, so each 20 points is 10× faster (median under the numbers). The dashed line is 2× Jev’s median latency.

Capability vs speed: 55 systems. Upper right is best: more capable and faster.10203040506070809010070≈3.2 s80≈1.0 s90≈316 ms100≈100 msMedian-latency speedCapability↑2× Jev latency← slowerfaster →1. Jev 1.13.02. Winnow-12B Q83. Cygnet4. Surogate Rune 26B-A4B v35. Jev-Omni
55 systems. Tap a bubble for its values.
  • Jev (TypeSafe, closed)
  • Jev rebuild
  • Instruction model, JSON schema
  • Small tool-calling model
  • Zero-shot classifier
  • Closed decision API
  • Reranker (neutral adapter)
  • Raw-logit control (base model)
  • Native-logit decision engine
  • Unclassified
  • system-one-open
  • faint = outside Jev-class

What-If: the weight sliders below re-score every system under other axis weights — only the equal 25/25/25/25 weights give the official option-A ranking. The 3D view of capability, cost and speed loads further down.

JevBench v1.5.1

JevBench Composite Score: 98 ranked systems

Official· four axes 0–100, equal-weight harmonic mean · Method notes ↓

Weights:
Adjust weights ↓
View by:Capability ↑

Cygnet and Winnow-12B Q8 are joint leaders (statistical tie).

Whiskers are 95% bootstrap intervals. 74 of 83 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other 14 adjacent pairs, so no tie classification is inferred.

Greener = stronger within its column.

98 of 98 systems, sorted by official rank, #1 first.

  1. 1Cygnet73.7I 71C 87S 91K 56est.$0.028
  2. 2Winnow-12B Q873.2I 74C 84S 86K 57est.$0.028
  3. 3Jev 1.13.0API72.1I 72C 88S 84K 55est.$0.032
  4. 4JevK5 v0.371.9I 56C 88S 94K 63est.$0.017
  5. 5Plumb-4B71.6I 56C 87S 93K 63est.$0.017
  6. 6Jev-Omni71.5I 70C 83S 85K 56est.$0.029
  7. 7decider-4b v271.3I 56C 86S 91K 65est.$0.015
  8. 8Decision 4B v1.270.8I 54C 89S 94K 63est.$0.017
  9. 9Imajev-4B70.4I 53C 88S 91K 63est.$0.017
  10. 10Decision 4B v1.170.4I 53C 87S 94K 63est.$0.017
  11. 11SemIf68.7I 51C 84S 91K 63est.$0.017
  12. 12spark-s1-4b-v668.2I 62C 70S 86K 60est.$0.021
  13. 13metask-jev-4b67.5I 53C 83S 89K 58est.$0.026
  14. 14Hopper67.5I 50C 88S 87K 62est.$0.018
  15. 15Malkuth-4B66.8I 54C 83S 86K 56est.$0.030
  16. 16Surogate Rune 26B-A4B v366.5I 70C 88S 86K 49est.$0.050
  17. 17reflex 4B65.2I 51C 87S 69K 63est.$0.017
  18. 18jev-local65.2I 56C 78S 73K 59est.$0.024
  19. 19djev (Maisa, diffusion-gemma)64.2I 72C 80S 91K 48est.$0.053
  20. 20Raw Qwen3 4B Instruct 2507 direct logits62.1I 54C 53S 89K 63est.$0.017
Show all 98 systems (78 more ranked, 0 more not ranked)
  1. 21jqv60.8I 49C 87S 83K 51est.$0.042
  2. 22JevK5 v0.2.058.1I 47C 85S 91K 63est.$0.017
  3. 23Qwen3.5-9B Jev-like data-mix v253.0I 60C 81S 82K 46est.$0.065
  4. 24Standard One 8B47.8I 60C 83S 92K 43est.$0.078
  5. 25NInfer Qwen3.8-Flash-Next mixed47.5I 67C 89S 89K 43est.$0.082
  6. 26swanOne46.6I 71C 87S 85K 42est.$0.085
  7. 27Raw Qwen3 8B direct logits45.2I 51C 49S 87K 46est.$0.065
  8. 28decider-2b45.1I 42C 72S 94K 65est.$0.015
  9. 29system-one44.1I 50C 49S 91K 45est.$0.068
  10. 30system-one-openAPI42.4I 42C 72S 78K 68est.$0.011
  11. 31Autoloops – Gemma 4 31B ITAPI40.5I 77C 86S 84K 40$0.103
  12. 32GPT-6 Luna (low reasoning effort)API40.5I 95C 95S 73K 39est.$0.108
  13. 33GPT-6 Luna (default medium reasoning effort)API38.8I 96C 96S 73K 38est.$0.114
  14. 34JevOne38.2I 54C 85S 90K 40est.$0.101
  15. 35kev 4B38.1I 40C 68S 85K 66est.$0.014
  16. 36kev 8B34.2I 48C 71S 84K 40est.$0.097
  17. 37open-alternative-jev33.6I 37C 77S 91K 63est.$0.017
  18. 38Bespoke Nimble 9B31.8I 64C 77S 83K 37est.$0.128
  19. 39Malkuth-2B29.9I 36C 75S 92K 66est.$0.014
  20. 40openjev-sglangAPI29.0I 59C 83S 78K 36est.$0.140
  21. 41decider-35b-a3b27.5I 60C 82S 91K 34est.$0.154
  22. 42local-jev Qwen3.5-4B25.8I 34C 82S 84K 59est.$0.023
  23. 43Open-Jev 9B24.4I 64C 82S 74K 33est.$0.170
  24. 44Decision 2B22.5I 31C 86S 90K 66est.$0.013
  25. 45GPT-5.6 LunaAPI22.4I 94C 95S 74K 31$0.205
  26. 46typecastlm21.8I 31C 77S 92K 64est.$0.016
  27. 47JEV Qwen3.5-9B Base NVFP420.1I 32C 81S 94K 48est.$0.056
  28. 48Gemini 3.1 Flash-LiteAPI19.6I 78C 75S 80K 30$0.219
  29. 49AutoJev-27B (denis-pplx, Qwen3.8-27B)19.5I 73C 88S 88K 29est.$0.226
  30. 50AutoJev-27B (RTX PRO 6000)19.5I 73C 87S 87K 29est.$0.226
  31. 51NInfer Qwen3.8-27B NVFP418.7I 65C 86S 90K 29est.$0.231
  32. 52Eikos-27B18.5I 75C 86S 88K 29est.$0.238
  33. 53NInfer Qwen3.8-27B NVFP4 (T=1.5)18.5I 61C 86S 90K 29est.$0.231
  34. 54InstinctAPI18.3I 63C 85S 82K 29est.$0.230
  35. 55OpenJev17.9I 84C 83S 74K 29est.$0.241
  36. 56djev (thinking)17.4I 77C 96S 72K 28est.$0.249
  37. 57LitJev16.3I 58C 84S 68K 28est.$0.244
  38. 58Raw Phi-4 mini direct logits15.2I 28C 71S 89K 53est.$0.036
  39. 59OpenSourceJev13.2I 26C 76S 73K 69est.$0.011
  40. 60reflex-27b13.2I 63C 86S 69K 26est.$0.297
  41. 61Open-Jev 2B9.1I 34C 74S 76K 33est.$0.170
  42. 62GLiNER2 large8.4I 22C 42S 65K 78est.$0.0056
  43. 63Qwen3-Reranker-4B7.1I 21C 76S 80K 48est.$0.052
  44. 64DeepSeek V4.1 FlashAPI6.6I 94C 97S 69K 19est.$0.498
  45. 65SimpleJev4.0I 17C 47S 59K 70est.$0.0098
  46. 66SimpleJev Qwen3.8-27BAPI3.4I 73C 87S 75K 15est.$0.687
  47. 67decision-machine-1API3.2I 15C 81S 93K 56est.$0.029
  48. 68GLiNER2.5 multi2.7I 14C 58S 67K 87est.$0.0028
  49. 69GLiNER22.3I 13C 36S 70K 87est.$0.0028
  50. 70JevActAPI1.5I 11C 63S 76K 69est.$0.011
  51. 71CLM-8B1.5I 11C 48S 93K 51est.$0.045
  52. 72kev 0.6B1.3I 10C 68S 87K 80est.$0.0046
  53. 73Raw Qwen3 0.6B direct logits1.1I 11C 21S 90K 78est.$0.0056
  54. 74GLiNER2.5 small0.9I 9C 56S 77K 87est.$0.0028
  55. 75Raw Qwen3 1.7B direct logits0.9I 10C 22S 90K 69est.$0.011
  56. 76Mirror0.2I 6C 43S 64K 89est.$0.0023
  57. 77ZeroEntropy zerank-20.1I 5C 82S 80K 48est.$0.052
  58. 78jeff0.1I 4C 80S 56K 81est.$0.0043
  59. 79smalljev semantic-v90.1I 4C 73S 86K 61est.$0.020
  60. 80OpenDecision0.0I 2C 73S 87K 79est.$0.0050
  61. 81BAAI bge-reranker-v2-m30.0I 0C 83S 91K 59est.$0.023
  62. 82Certo v10.0I 0C 88S 91K 97est.$0.0013
  63. 83Decision Fast0.0I 0C 76S 91K 80est.$0.0046
  64. 84Alibaba GTE Reranker ModernBERT-base0.0I 0C 75S 91K 69est.$0.011
  65. 85kev 0.5B0.0I 0C 65S 88K 80est.$0.0046
  66. 86Laya0.0I 0C 74S 74K 85est.$0.0032
  67. 87lev-350m0.0I 0C 78S 94K 80est.$0.0046
  68. 88Qwen3.5-0.8B Decision Model0.0I 0C 74S 72K 80est.$0.0048
  69. 89Mixedbread mxbai-rerank-base-v20.0I 0C 87S 89K 60est.$0.021
  70. 90Needle 30.0I 0C 0S 34K 62est.$0.019
  71. 91Needle 3, options as tools0.0I 0C 0S 41K 62est.$0.019
  72. 92open-jev-deberta-v3-large0.0I 0C 77S 68K 78est.$0.0056
  73. 93Open Jev JSON Canvas0.0I 77C 0S 86K 49est.$0.049
  74. 94openJev Verdict0.0I 0C 52S 84K 87est.$0.0028
  75. 95openJev Verdict 1.40.0I 0C 80S 81K 87est.$0.0028
  76. 96Qwen3.8 27BAPI0.0I 96C 98S 57K 0est.$2.178
  77. 97verdict-small0.0I 0C 59S 82K 100est.$0.0009
  78. 98Von0.0I 0C 83S 76K 83est.$0.0038
Weights:
Adjust weights ↓

Weights are relative: each axis counts in proportion to its slider. The score stays a weighted harmonic mean with the low-axis gates; the gates still apply when an axis sits at 0. Only equal weights give the official JevBench Score and rank.

Score = 4 / (1/I + 1/C + 1/S + 1/K) (each 0–100; × (axis / 50)² for Intelligence, Speed or Cost below 50)

  • Jev (TypeSafe, closed)
  • Jev rebuild
  • Instruction model, JSON schema
  • Small tool-calling model
  • Zero-shot classifier
  • Closed decision API
  • Reranker (neutral adapter)
  • Raw-logit control (base model)
  • Native-logit decision engine
  • Unclassified
  • system-one-open
I, C, S, K = Intelligence, Calibration, Speed, Cost; the est. pill = estimated cost; ann. = announced price; API = the operator's endpoint saw sealed item text, without answers; $/1k decisions = US dollars per 1,000 decisions (not heat-shaded). Names link to each project.

Honorable mentions and Jev wrappers — listed separately, not ranked (1)

Eligibility rule: services that run on Jev itself may be measured and shown as honorable mentions, but are not competitors ranked against Jev and do not enter the field median gap (G_med) or tie markers. classifier.dev (TypeSafe) runs on Jev, so it stays unranked under this rule.

  • classifier.dev (fast tier)APIhonorable mention: runs on Jev (TypeSafe) - listed, not ranked (honorable mention, as in v1.4.2). Official (A) score 74.7.

Compare two systems

Pick any two. Four radars: the score axes, chance-corrected competence per request type on the open and sealed sets, and competence per tier on each set. Further out is better on every spoke; the link keeps the pair.

  • A: Jev 1.13.0 — Jev (TypeSafe, closed) · Score 72.1 (#3)
  • B: Cygnet — system-one-open · Score 73.7 (#1)

The four score axes

Radar: the four score axes, two systemsThe four score axes, Jev 1.13.0 vs Cygnet. Intelligence: 72.0 vs 71.1; Calibration: 88.0 vs 87.0; Speed: 83.8 vs 91.0; Cost: 54.7 vs 56.4.50100Intelligence72.0 · 71.1Calibration88.0 · 87.0Speed83.8 · 91.0Cost54.7 · 56.4
0–100, the values in the table. A system with no published axis draws at 0 and says so.

Competence per request type, open / sealed

Radar: competence per request type, open / sealed, two systemsCompetence per request type, open / sealed, Jev 1.13.0 vs Cygnet. Choice · open: 85.7 vs 82.9; Choice · sealed: 87.6 vs 80.3; Noul · open: 47.8 vs 57.9; Noul · sealed: 48.6 vs 55.7; Score · open: 81.2 vs 76.6; Score · sealed: 81.1 vs 73.2.50100Choice · open85.7 · 82.9Choice ·sealed87.6 · 80.3Noul · open47.8 · 57.9Noul · sealed48.6 · 55.7Score · open81.2 · 76.6Score ·sealed81.1 · 73.2
Chance-corrected competence (0 = chance) for Choice, Noul and Score on the 904 open and 720 sealed decisions.

Competence per tier — open set

Radar: competence per tier — open set, two systemsCompetence per tier — open set, Jev 1.13.0 vs Cygnet. Easy: 90.0 vs 89.7; Standard: 80.9 vs 82.4; Judge: 83.5 vs 82.2; Hard: 53.8 vs 58.0.50100Easy90.0 · 89.7Standard80.9 · 82.4Judge83.5 · 82.2Hard53.8 · 58.0
Per-tier competence, the three request types pooled by their published decision counts.

Competence per tier — sealed set

Radar: competence per tier — sealed set, two systemsCompetence per tier — sealed set, Jev 1.13.0 vs Cygnet. Easy: 91.2 vs 88.6; Standard: 79.0 vs 73.5; Judge: 73.7 vs 75.4; Hard: 72.9 vs 65.3.50100Easy91.2 · 88.6Standard79.0 · 73.5Judge73.7 · 75.4Hard72.9 · 65.3
Per-tier competence on the sealed decisions, types pooled the same way; item text stays private.
All values as a table
SpokeA: Jev 1.13.0B: Cygnet
The four score axes
Intelligence72.071.1
Calibration88.087.0
Speed83.891.0
Cost54.756.4
Competence per request type, open / sealed
Choice · open85.782.9
Choice · sealed87.680.3
Noul · open47.857.9
Noul · sealed48.655.7
Score · open81.276.6
Score · sealed81.173.2
Competence per tier — open set
Easy90.089.7
Standard80.982.4
Judge83.582.2
Hard53.858.0
Competence per tier — sealed set
Easy91.288.6
Standard79.073.5
Judge73.775.4
Hard72.965.3

Axes, request types, latency and cost

Compare:

Every measured system, under the four score axes. Intelligence is 50% open (904 decisions) and 50% sealed (720); Gap = I_open − I_sealed, and the penalty applies only above the field median gap (G_med 5.2) plus 8. The per-type competence, latency and price of the same systems are in Types & cost. On a phone the name column stays put while the table scrolls sideways.

#ASystemScoreIntel.Calib.SpeedCostGapPenalty
1Cygnet73.771.187.091.056.4+2.7×1.000
2Winnow-12B Q873.274.484.186.156.6-3.9×1.000
3Jev 1.13.0API72.172.088.083.854.7-0.9×1.000
4JevK5 v0.3v1.5 roster addendum A171.956.388.393.663.1+10.7×1.000
5Plumb-4Bv1.5 roster addendum A171.655.887.493.563.1+10.1×1.000
6Jev-Omni71.570.582.684.756.1-0.0×1.000
7decider-4b v271.355.885.690.964.5+11.9×1.000
8Decision 4B v1.2v1.5 roster addendum A170.853.788.693.563.1+5.2×1.000
9Imajev-4Bv1.5 roster addendum A270.453.588.191.163.3+5.5×1.000
10Decision 4B v1.1v1.5 roster addendum A170.453.187.393.563.1+3.1×1.000
11SemIf68.751.384.090.963.1+5.4×1.000
12spark-s1-4b-v668.262.169.785.560.4+0.7×1.000
13metask-jev-4b67.553.582.789.557.7+6.5×1.000
14Hopper67.549.987.987.262.3-2.3×1.000
15Malkuth-4B66.854.583.286.255.8+7.8×1.000
16Surogate Rune 26B-A4B v3v1.5 roster addendum A266.569.788.386.049.0-1.6×1.000
17reflex 4B65.251.586.868.863.1+7.4×1.000
18jev-local65.256.377.873.058.9+3.8×1.000
19djev (Maisa, diffusion-gemma)64.272.380.491.048.2+0.8×1.000
20Raw Qwen3 4B Instruct 2507 direct logits62.154.152.989.263.4+6.7×1.000
21jqv60.849.186.883.451.2+4.5×1.000
22JevK5 v0.2.058.146.784.990.963.1+6.6×1.000
23Qwen3.5-9B Jev-like data-mix v253.060.480.982.045.7+4.3×1.000
24Standard One 8B47.859.683.292.543.3+0.7×1.000
25NInfer Qwen3.8-Flash-Next mixed47.567.288.588.642.5+6.1×1.000
26swanOne46.671.287.184.742.2-1.5×1.000
27Raw Qwen3 8B direct logits45.251.149.286.845.5+11.3×1.000
28decider-2b45.142.371.594.464.9+12.9×1.000
29system-one44.150.549.490.945.0+6.1×1.000
30system-one-openAPI42.441.671.678.368.3+5.6×1.000
31Autoloops – Gemma 4 31B ITAPI40.576.785.883.939.6+0.4×1.000
32GPT-6 Luna (low reasoning effort)API40.595.394.973.239.1-1.1×1.000
33GPT-6 Luna (default medium reasoning effort)API38.896.295.673.238.3+0.2×1.000
34JevOne38.254.184.990.439.8-0.5×1.000
35kev 4B38.139.967.685.565.8+13.9×0.993
36kev 8B34.248.371.084.140.4+12.0×1.000
37open-alternative-jev33.637.476.991.363.3+10.5×1.000
38Bespoke Nimble 9B31.863.777.283.036.8+2.8×1.000
39Malkuth-2B29.935.575.291.865.6+15.6×0.976
40openjev-sglangAPI29.058.682.978.135.6+5.4×1.000
41decider-35b-a3b27.560.582.091.034.4+7.4×1.000
42local-jev Qwen3.5-4B25.833.782.483.759.3+5.9×1.000
43Open-Jev 9B24.463.881.573.633.1-4.1×1.000
44Decision 2B22.531.386.290.466.4+6.7×1.000
45GPT-5.6 LunaAPI22.494.394.774.130.7-2.3×1.000
46typecastlm21.831.276.792.064.0-15.1×1.000
47JEV Qwen3.5-9B Base NVFP420.132.280.993.547.6+1.3×1.000
48Gemini 3.1 Flash-LiteAPI19.677.674.780.129.8-1.9×1.000
49AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A119.572.887.787.629.4-1.7×1.000
50AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A219.572.886.787.129.4-1.5×1.000
51NInfer Qwen3.8-27B NVFP418.765.585.989.929.1+3.0×1.000
52Eikos-27Bv1.5 roster addendum A118.575.186.387.628.7-1.9×1.000
53NInfer Qwen3.8-27B NVFP4 (T=1.5)18.561.186.589.929.1+2.7×1.000
54InstinctAPI18.362.785.081.729.2-1.7×1.000
55OpenJev (thinking, BF16)17.984.283.173.928.5-3.4×1.000
56djev (thinking)17.477.395.772.328.1-1.6×1.000
57LitJev16.358.384.568.228.4-0.5×1.000
58Raw Phi-4 mini direct logits15.227.671.489.253.2+18.3×0.949
59OpenSourceJev13.225.876.072.569.1+4.2×1.000
60reflex-27b13.262.885.969.225.8+0.6×1.000
61Open-Jev 2B9.133.673.675.833.1+10.4×1.000
62GLiNER2 large8.422.542.464.877.6+7.1×1.000
63Qwen3-Reranker-4B7.121.076.379.648.4+17.0×0.962
64DeepSeek V4.1 FlashAPI6.693.796.969.419.1-3.7×1.000
65SimpleJev4.016.746.759.170.3-6.8×1.000
66SimpleJev Qwen3.8-27BAPI3.472.887.274.914.9-1.7×1.000
67decision-machine-1API3.214.881.292.856.3+22.4×0.908
68GLiNER2.5 multi2.714.058.066.686.6+3.6×1.000
69GLiNER22.313.535.670.386.6+7.5×1.000
70JevActAPI1.511.262.576.468.6+16.3×0.969
71CLM-8B1.511.448.593.350.5+11.3×1.000
72kev 0.6B1.310.367.787.280.1+27.0×0.862
73Raw Qwen3 0.6B direct logits1.110.621.590.477.6-1.1×1.000
74GLiNER2.5 small0.99.255.977.486.6+15.5×0.977
75Raw Qwen3 1.7B direct logits0.99.721.690.268.5+5.2×1.000
76Mirror0.25.643.263.689.3+2.3×1.000
77ZeroEntropy zerank-20.14.881.780.348.4+10.4×1.000
78jeff0.14.480.355.881.1+16.3×0.969
79smalljev semantic-v90.13.773.186.560.8+14.5×0.987
80OpenDecision0.02.172.586.679.1+14.7×0.985
81BAAI bge-reranker-v2-m30.00.083.390.559.3+1.7×1.000
82Certo v10.00.087.591.496.9+0.8×1.000
83Decision Fast0.00.076.191.480.1+19.0×0.942
84Alibaba GTE Reranker ModernBERT-base0.00.075.291.569.3+4.5×1.000
85kev 0.5B0.00.065.088.580.1+25.7×0.875
86Laya0.00.073.773.984.9+12.7×1.000
87lev-350m0.00.077.793.680.0+23.5×0.896
88Qwen3.5-0.8B Decision Model0.00.073.671.879.5+7.1×1.000
89Mixedbread mxbai-rerank-base-v20.00.086.689.360.3+3.0×1.000
90Needle 30.00.00.034.161.6+10.0×1.000
91Needle 3, options as tools0.00.00.041.061.6+14.1×0.991
92open-jev-deberta-v3-large0.00.077.168.377.6+5.3×1.000
93Open Jev JSON Canvas0.077.10.085.649.3+4.0×1.000
94openJev Verdict0.00.052.283.986.6+26.7×0.865
95openJev Verdict 1.40.00.080.380.686.6+3.5×1.000
96Qwen3.8 27BAPI0.095.698.156.80.0-5.3×1.000
97verdict-small0.00.059.582.1100.0+2.0×1.000
98Von0.00.083.575.782.7+14.7×0.985
–classifier.devAPIhonorable mention—75.889.282.058.9-2.7×1.000
–SimpleJev Qwen3.6-35B-A3BAPIpartial run—0.078.575.235.1+5.6×1.000
Official order with 95% intervals (98 systems)

JevBench v1.5.1 · headline option A

JevBench Score: 98 ranked systems

Official (A)weighted harmonic mean of four 0–100 axes, Intelligence · Calibration · Speed · Cost = 25 · 25 · 25 · 25, with the low-axis gates · Method ↓

Cygnet and Winnow-12B Q8 are joint leaders (statistical tie).

Whiskers are 95% bootstrap intervals. 74 of 83 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other 14 adjacent pairs, so no tie classification is inferred.

  1. 1Cygnet73.7I 71 · C 87 · S 91 · K 56 · B#2 · C#1 · $0.028
  2. 2Winnow-12B Q873.2I 74 · C 84 · S 86 · K 57 · B#1 · C#2 · $0.028
  3. 3Jev 1.13.0API72.1I 72 · C 88 · S 84 · K 55 · B#3 · C#3 · $0.032
  4. 4JevK5 v0.3v1.5 roster addendum A171.9I 56 · C 88 · S 94 · K 63 · B#5 · C#8 · $0.017
  5. 5Plumb-4Bv1.5 roster addendum A171.6I 56 · C 87 · S 93 · K 63 · B#6 · C#9 · $0.017
  6. 6Jev-Omni71.5I 70 · C 83 · S 85 · K 56 · B#4 · C#4 · $0.029
  7. 7decider-4b v271.3I 56 · C 86 · S 91 · K 65 · B#7 · C#10 · $0.015
  8. 8Decision 4B v1.2v1.5 roster addendum A170.8I 54 · C 89 · S 94 · K 63 · B#9 · C#12 · $0.017
  9. 9Imajev-4Bv1.5 roster addendum A270.4I 53 · C 88 · S 91 · K 63 · B#11 · C#13 · $0.017
  10. 10Decision 4B v1.1v1.5 roster addendum A170.4I 53 · C 87 · S 94 · K 63 · B#12 · C#14 · $0.017
  11. 11SemIf68.7I 51 · C 84 · S 91 · K 63 · B#14 · C#19 · $0.017
  12. 12spark-s1-4b-v668.2I 62 · C 70 · S 86 · K 60 · B#8 · C#5 · $0.021
  13. 13metask-jev-4b67.5I 53 · C 83 · S 89 · K 58 · B#15 · C#16 · $0.026
  14. 14Hopper67.5I 50 · C 88 · S 87 · K 62 · B#18 · C#23 · $0.018
  15. 15Malkuth-4B66.8I 54 · C 83 · S 86 · K 56 · B#16 · C#15 · $0.030
  16. 16Surogate Rune 26B-A4B v3v1.5 roster addendum A266.5I 70 · C 88 · S 86 · K 49 · B#10 · C#6 · $0.050
  17. 17reflex 4B65.2I 51 · C 87 · S 69 · K 63 · B#19 · C#20 · $0.017
  18. 18jev-local65.2I 56 · C 78 · S 73 · K 59 · B#17 · C#11 · $0.024
  19. 19djev (Maisa, diffusion-gemma)64.2I 72 · C 80 · S 91 · K 48 · B#13 · C#7 · $0.053
  20. 20Raw Qwen3 4B Instruct 2507 direct logits62.1I 54 · C 53 · S 89 · K 63 · B#20 · C#18 · $0.017
  21. 21jqv60.8I 49 · C 87 · S 83 · K 51 · B#21 · C#25 · $0.042
  22. 22JevK5 v0.2.058.1I 47 · C 85 · S 91 · K 63 · B#22 · C#28 · $0.017
  23. 23Qwen3.5-9B Jev-like data-mix v253.0I 60 · C 81 · S 82 · K 46 · B#23 · C#17 · $0.065
  24. 24Standard One 8B47.8I 60 · C 83 · S 92 · K 43 · B#26 · C#22 · $0.078
  25. 25NInfer Qwen3.8-Flash-Next mixed47.5I 67 · C 89 · S 89 · K 43 · B#24 · C#21 · $0.082
  26. 26swanOne46.6I 71 · C 87 · S 85 · K 42 · B#25 · C#24 · $0.085
  27. 27Raw Qwen3 8B direct logits45.2I 51 · C 49 · S 87 · K 46 · B#27 · C#30 · $0.065
  28. 28decider-2b45.1I 42 · C 72 · S 94 · K 65 · B#32 · C#32 · $0.015
  29. 29system-one44.1I 50 · C 49 · S 91 · K 45 · B#28 · C#33 · $0.068
  30. 30system-one-openAPI42.4I 42 · C 72 · S 78 · K 68 · B#33 · C#35 · $0.011
  31. 31Autoloops – Gemma 4 31B ITAPI40.5I 77 · C 86 · S 84 · K 40 · B#30 · C#26 · tariff$0.103
  32. 32GPT-6 Luna (low reasoning effort)API40.5I 95 · C 95 · S 73 · K 39 · B#29 · C#27 · $0.108
  33. 33GPT-6 Luna (default medium reasoning effort)API38.8I 96 · C 96 · S 73 · K 38 · B#31 · C#29 · $0.114
  34. 34JevOne38.2I 54 · C 85 · S 90 · K 40 · B#34 · C#34 · $0.101
  35. 35kev 4B38.1I 40 · C 68 · S 85 · K 66 · B#35 · C#38 · $0.014
  36. 36kev 8B34.2I 48 · C 71 · S 84 · K 40 · B#36 · C#40 · $0.097
  37. 37open-alternative-jev33.6I 37 · C 77 · S 91 · K 63 · B#38 · C#41 · $0.017
  38. 38Bespoke Nimble 9B31.8I 64 · C 77 · S 83 · K 37 · B#37 · C#31 · $0.128
  39. 39Malkuth-2B29.9I 36 · C 75 · S 92 · K 66 · B#41 · C#43 · $0.014
  40. 40openjev-sglangAPI29.0I 59 · C 83 · S 78 · K 36 · B#39 · C#36 · $0.140
  41. 41decider-35b-a3b27.5I 60 · C 82 · S 91 · K 34 · B#40 · C#37 · $0.154
  42. 42local-jev Qwen3.5-4B25.8I 34 · C 82 · S 84 · K 59 · B#44 · C#52 · $0.023
  43. 43Open-Jev 9B24.4I 64 · C 82 · S 74 · K 33 · B#42 · C#39 · $0.170
  44. 44Decision 2B22.5I 31 · C 86 · S 90 · K 66 · B#50 · C#54 · $0.013
  45. 45GPT-5.6 LunaAPI22.4I 94 · C 95 · S 74 · K 31 · B#43 · C#42 · tariff$0.205
  46. 46typecastlm21.8I 31 · C 77 · S 92 · K 64 · B#54 · C#56 · $0.016
  47. 47JEV Qwen3.5-9B Base NVFP420.1I 32 · C 81 · S 94 · K 48 · B#56 · C#57 · $0.056
  48. 48Gemini 3.1 Flash-LiteAPI19.6I 78 · C 75 · S 80 · K 30 · B#45 · C#44 · tariff$0.219
  49. 49AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A119.5I 73 · C 88 · S 88 · K 29 · B#46 · C#45 · $0.226
  50. 50AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A219.5I 73 · C 87 · S 87 · K 29 · B#47 · C#46 · $0.226
  51. 51NInfer Qwen3.8-27B NVFP418.7I 65 · C 86 · S 90 · K 29 · B#49 · C#47 · $0.231
  52. 52Eikos-27Bv1.5 roster addendum A118.5I 75 · C 86 · S 88 · K 29 · B#48 · C#48 · $0.238
  53. 53NInfer Qwen3.8-27B NVFP4 (T=1.5)18.5I 61 · C 86 · S 90 · K 29 · B#52 · C#49 · $0.231
  54. 54InstinctAPI18.3I 63 · C 85 · S 82 · K 29 · B#53 · C#50 · $0.230
  55. 55OpenJev (thinking, BF16)17.9I 84 · C 83 · S 74 · K 29 · B#51 · C#51 · $0.241
  56. 56djev (thinking)17.4I 77 · C 96 · S 72 · K 28 · B#55 · C#53 · $0.249
  57. 57LitJev16.3I 58 · C 84 · S 68 · K 28 · B#57 · C#55 · $0.244
  58. 58Raw Phi-4 mini direct logits15.2I 28 · C 71 · S 89 · K 53 · B#59 · C#59 · $0.036
  59. 59OpenSourceJev13.2I 26 · C 76 · S 73 · K 69 · B#60 · C#60 · $0.011
  60. 60reflex-27b13.2I 63 · C 86 · S 69 · K 26 · B#58 · C#58 · $0.297
  61. 61Open-Jev 2B9.1I 34 · C 74 · S 76 · K 33 · B#61 · C#62 · $0.170
  62. 62GLiNER2 large8.4I 22 · C 42 · S 65 · K 78 · B#63 · C#63 · $0.0056
  63. 63Qwen3-Reranker-4B7.1I 21 · C 76 · S 80 · K 48 · B#64 · C#64 · $0.052
  64. 64DeepSeek V4.1 FlashAPI6.6I 94 · C 97 · S 69 · K 19 · B#62 · C#61 · $0.498
  65. 65SimpleJev4.0I 17 · C 47 · S 59 · K 70 · B#66 · C#66 · $0.0098
  66. 66SimpleJev Qwen3.8-27BAPI3.4I 73 · C 87 · S 75 · K 15 · B#65 · C#65 · $0.687
  67. 67decision-machine-1API3.2I 15 · C 81 · S 93 · K 56 · B#67 · C#67 · $0.029
  68. 68GLiNER2.5 multi2.7I 14 · C 58 · S 67 · K 87 · B#68 · C#68 · $0.0028
  69. 69GLiNER22.3I 13 · C 36 · S 70 · K 87 · B#69 · C#69 · $0.0028
  70. 70JevActAPI1.5I 11 · C 63 · S 76 · K 69 · B#71 · C#70 · $0.011
  71. 71CLM-8B1.5I 11 · C 48 · S 93 · K 51 · B#70 · C#71 · $0.045
  72. 72kev 0.6B1.3I 10 · C 68 · S 87 · K 80 · B#72 · C#72 · $0.0046
  73. 73Raw Qwen3 0.6B direct logits1.1I 11 · C 21 · S 90 · K 78 · B#73 · C#73 · $0.0056
  74. 74GLiNER2.5 small0.9I 9 · C 56 · S 77 · K 87 · B#75 · C#74 · $0.0028
  75. 75Raw Qwen3 1.7B direct logits0.9I 10 · C 22 · S 90 · K 69 · B#74 · C#75 · $0.011
  76. 76Mirror0.2I 6 · C 43 · S 64 · K 89 · B#76 · C#76 · $0.0023
  77. 77ZeroEntropy zerank-20.1I 5 · C 82 · S 80 · K 48 · B#77 · C#77 · $0.052
  78. 78jeff0.1I 4 · C 80 · S 56 · K 81 · B#78 · C#78 · $0.0043
  79. 79smalljev semantic-v90.1I 4 · C 73 · S 86 · K 61 · B#79 · C#79 · $0.020
  80. 80OpenDecision0.0I 2 · C 73 · S 87 · K 79 · B#80 · C#80 · $0.0050
  81. 81BAAI bge-reranker-v2-m30.0I 0 · C 83 · S 91 · K 59 · B#81 · C#81 · $0.023
  82. 82Certo v10.0I 0 · C 88 · S 91 · K 97 · B#82 · C#82 · $0.0013
  83. 83Decision Fast0.0I 0 · C 76 · S 91 · K 80 · B#83 · C#83 · $0.0046
  84. 84Alibaba GTE Reranker ModernBERT-base0.0I 0 · C 75 · S 91 · K 69 · B#84 · C#84 · $0.011
  85. 85kev 0.5B0.0I 0 · C 65 · S 88 · K 80 · B#85 · C#85 · $0.0046
  86. 86Laya0.0I 0 · C 74 · S 74 · K 85 · B#86 · C#86 · $0.0032
  87. 87lev-350m0.0I 0 · C 78 · S 94 · K 80 · B#87 · C#87 · $0.0046
  88. 88Qwen3.5-0.8B Decision Model0.0I 0 · C 74 · S 72 · K 80 · B#88 · C#88 · $0.0048
  89. 89Mixedbread mxbai-rerank-base-v20.0I 0 · C 87 · S 89 · K 60 · B#89 · C#89 · $0.021
  90. 90Needle 30.0I 0 · C 0 · S 34 · K 62 · B#90 · C#90 · $0.019
  91. 91Needle 3, options as tools0.0I 0 · C 0 · S 41 · K 62 · B#91 · C#91 · $0.019
  92. 92open-jev-deberta-v3-large0.0I 0 · C 77 · S 68 · K 78 · B#92 · C#92 · $0.0056
  93. 93Open Jev JSON Canvas0.0I 77 · C 0 · S 86 · K 49 · B#93 · C#93 · $0.049
  94. 94openJev Verdict0.0I 0 · C 52 · S 84 · K 87 · B#94 · C#94 · $0.0028
  95. 95openJev Verdict 1.40.0I 0 · C 80 · S 81 · K 87 · B#95 · C#95 · $0.0028
  96. 96Qwen3.8 27BAPI0.0I 96 · C 98 · S 57 · K 0 · B#96 · C#96 · $2.18
  97. 97verdict-small0.0I 0 · C 59 · S 82 · K 100 · B#97 · C#97 · $0.0009
  98. 98Von0.0I 0 · C 83 · S 76 · K 83 · B#98 · C#98 · $0.0038

Costs are estimates (est.) unless marked tariff.

system-one-openJev rebuildJev (TypeSafe, closed)UnclassifiedRaw-logit control (base model)Native-logit decision engineInstruction model, JSON schemaZero-shot classifierReranker (neutral adapter)Closed decision APISmall tool-calling model

All three weight options (98 systems)

All three weight options

A is the official headline: equal 25/25/25/25 axis weights and an Intelligence floor of 50. B remains the secondary 40/20/20/20 axis-weight view; C retains equal axes with an Intelligence floor of 60. All three use equal Choice/Noul/Score weights. The CI column is the paired-bootstrap 95% interval of the A score.

#ASystemA · equal (headline)B · 40/20/20/20C · equal, I floor 60#B#CA 95% CI
1Cygnet73.773.273.72172.4–74.5
2Winnow-12B Q873.273.573.21272.0–74.0
3Jev 1.13.0API72.172.172.13371.0–72.6
4JevK5 v0.3v1.5 roster addendum A171.968.163.25869.4–72.9
5Plumb-4Bv1.5 roster addendum A171.667.762.06969.2–72.7
6Jev-Omni71.571.371.54470.2–72.4
7decider-4b v271.367.561.671069.1–72.3
8Decision 4B v1.2v1.5 roster addendum A170.866.656.791268.5–72.0
9Imajev-4Bv1.5 roster addendum A270.466.255.9111367.8–71.6
10Decision 4B v1.1v1.5 roster addendum A170.466.155.1121466.9–71.6
11SemIf68.764.350.2141960.1–70.2
12spark-s1-4b-v668.266.968.28566.3–69.7
13metask-jev-4b67.564.153.6151665.4–68.7
14Hopper67.562.946.9182356.3–69.2
15Malkuth-4B66.863.955.0161564.9–67.9
16Surogate Rune 26B-A4B v3v1.5 roster addendum A266.566.666.510665.4–67.0
17reflex 4B65.261.948.0192058.6–66.3
18jev-local65.263.257.4171163.5–66.5
19djev (Maisa, diffusion-gemma)64.264.864.213763.1–65.0
20Raw Qwen3 4B Instruct 2507 direct logits62.160.350.4201859.4–64.2
21jqv60.857.542.2212551.3–64.3
22JevK5 v0.2.058.153.640.4222847.6–68.2
23Qwen3.5-9B Jev-like data-mix v253.052.553.0231751.9–53.9
24Standard One 8B47.847.147.2262246.7–48.5
25NInfer Qwen3.8-Flash-Next mixed47.547.747.5242146.7–47.9
26swanOne46.647.346.6252445.8–46.9
27Raw Qwen3 8B direct logits45.244.632.9273036.9–46.7
28decider-2b45.141.131.3323233.8–54.3
29system-one44.143.431.2283336.8–45.7
30system-one-openAPI42.438.829.5333533.8–51.9
31Autoloops – Gemma 4 31B ITAPI40.541.840.5302640.1–40.8
32GPT-6 Luna (low reasoning effort)API40.543.140.5292740.3–40.7
33GPT-6 Luna (default medium reasoning effort)API38.841.438.8312938.6–38.9
34JevOne38.237.431.1343437.4–38.8
35kev 4B38.134.626.4353827.5–46.8
36kev 8B34.233.123.7364026.8–37.5
37open-alternative-jev33.629.923.3384125.8–41.7
38Bespoke Nimble 9B31.832.331.8373131.2–32.3
39Malkuth-2B29.926.420.8414321.0–39.0
40openjev-sglangAPI29.029.227.7393628.4–29.4
41decider-35b-a3b27.527.727.5403726.9–27.9
42local-jev Qwen3.5-4B25.822.717.9445219.9–33.0
43Open-Jev 9B24.425.024.4423923.9–24.6
44Decision 2B22.519.315.6505416.7–28.8
45GPT-5.6 LunaAPI22.424.222.4434222.3–22.5
46typecastlm21.818.915.2545616.2–28.6
47JEV Qwen3.5-9B Base NVFP420.117.813.9565715.2–25.5
48Gemini 3.1 Flash-LiteAPI19.620.819.6454419.3–19.8
49AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A119.520.419.5464519.3–19.6
50AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A219.520.419.5474619.3–19.6
51NInfer Qwen3.8-27B NVFP418.719.318.7494718.5–18.9
52Eikos-27Bv1.5 roster addendum A118.519.518.5484818.3–18.6
53NInfer Qwen3.8-27B NVFP4 (T=1.5)18.518.918.5524918.2–18.6
54InstinctAPI18.318.918.3535018.0–18.5
55OpenJev (thinking, BF16)17.919.317.9515117.8–18.1
56djev (thinking)17.418.517.4555317.3–17.5
57LitJev16.316.715.4575516.0–16.5
58Raw Phi-4 mini direct logits15.213.110.6595910.3–20.7
59OpenSourceJev13.211.29.260609.0–18.3
60reflex-27b13.213.813.2585813.0–13.3
61Open-Jev 2B9.18.46.361626.6–11.5
62GLiNER2 large8.47.25.863635.2–12.3
63Qwen3-Reranker-4B7.15.94.964644.3–10.7
64DeepSeek V4.1 FlashAPI6.67.46.662616.6–6.7
65SimpleJev4.03.32.866662.1–6.7
66SimpleJev Qwen3.8-27BAPI3.43.73.465653.3–3.4
67decision-machine-1API3.22.52.367671.7–5.4
68GLiNER2.5 multi2.72.11.968681.2–5.1
69GLiNER22.31.81.669690.9–4.5
70JevActAPI1.51.11.171700.5–3.2
71CLM-8B1.51.21.070710.5–3.2
72kev 0.6B1.30.90.972720.5–2.5
73Raw Qwen3 0.6B direct logits1.10.90.773730.3–2.6
74GLiNER2.5 small0.90.60.675740.2–2.2
75Raw Qwen3 1.7B direct logits0.90.70.674750.1–2.3
76Mirror0.20.20.276760.0–0.9
77ZeroEntropy zerank-20.10.10.177770.0–0.5
78jeff0.10.10.178780.0–0.5
79smalljev semantic-v90.10.00.079790.0–0.4
80OpenDecision0.00.00.080800.0–0.2
81BAAI bge-reranker-v2-m30.00.00.081810.0–0.0
82Certo v10.00.00.082820.0–0.0
83Decision Fast0.00.00.083830.0–0.0
84Alibaba GTE Reranker ModernBERT-base0.00.00.084840.0–0.0
85kev 0.5B0.00.00.085850.0–0.0
86Laya0.00.00.086860.0–0.0
87lev-350m0.00.00.087870.0–0.0
88Qwen3.5-0.8B Decision Model0.00.00.088880.0–0.0
89Mixedbread mxbai-rerank-base-v20.00.00.089890.0–0.0
90Needle 30.00.00.090900.0–0.0
91Needle 3, options as tools0.00.00.091910.0–0.0
92open-jev-deberta-v3-large0.00.00.092920.0–0.0
93Open Jev JSON Canvas0.00.00.093930.0–0.0
94openJev Verdict0.00.00.094940.0–0.0
95openJev Verdict 1.40.00.00.095950.0–0.0
96Qwen3.8 27BAPI0.00.00.096960.0–0.0
97verdict-small0.00.00.097970.0–0.0
98Von0.00.00.098980.0–0.0

What the run says

  • Among the 55 Jev-class systems, Jev 1.13.0 has the highest Capability, 80.0 (Intelligence 72.0, Calibration 88.0).
  • Cygnet leads the official JevBench v1.5.1 score (option A) with 73.7: Intelligence 71.1, Calibration 87.0, Speed 91.0, Cost 56.4 ($0.028 per 1,000 decisions).
  • The best open or open-planned rebuild, Winnow-12B Q8, is #2 at 73.2 — 0.5 points behind.
  • GPT-6 Luna (default medium reasoning effort) has the highest Intelligence (96.2) but places #33: Speed 73.2, Cost 38.3 — the harmonic mean does not let accuracy buy back a weak axis.
  • The strongest sealed Intelligence is 98.2 (Qwen3.8 27B, #96); sealed items carry half of Intelligence, and an open-minus-sealed gap beyond the field median plus eight points costs Intelligence.
  • 74 of 83 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other adjacent pairs, so no tie classification is inferred.
  • All 9 systems joined by separately hashed roster addenda and have official ranks in this revision; their A/B/C ranks and intervals are in the addendum table below.
  • classifier.dev scores 74.7 but is not ranked: runs on Jev (TypeSafe) - listed, not ranked (honorable mention, as in v1.4.2).
  • SimpleJev Qwen3.6-35B-A3B did not complete the full suite; they are listed without a rank.

Jev alternatives, open source and self-hosting

The chart and table above compare the tested systems, not marketing claims. These are the practical answers readers most often need before choosing a Jev-class decision model.

What are open-source alternatives to Jev?

The highest-ranked open entrants in this run are Cygnet (#1, 73.7), Winnow-12B Q8 (#2, 73.2), JevK5 v0.3 (#4, 71.9), Plumb-4B (#5, 71.6). “Open” here means the tested row publishes code or weights; check the licence and exact configuration in the board before adopting one.

Which Jev-class models can I self-host in the EU or use for GDPR-sensitive work?

Open entrants with released code or weights can run on infrastructure you choose, including EU infrastructure. That can support data residency, but neither open source nor an EU server makes a deployment GDPR-compliant by itself. Assess your data, contracts, retention, subprocessors and security for the complete setup. See Benchmark Heaven's broader EU-hosting comparison.

jev-router.com offers self-hosted open decision models. Neutrality disclosure: it is run by the authors of this benchmark; it receives no scoring advantage and is not a ranked entrant.

How is JevBench scored?

The official score (option A) is the equal-weight harmonic mean of Intelligence, Calibration, Speed and Cost — 25/25/25/25 — with an Intelligence floor of 50 and low-axis gates on Speed and Cost. Version v1.5.1 measures 904 open and 720 sealed decisions per system; Choice, Noul and Score each carry a third, sealed items contribute 50% of Intelligence, and an open-minus-sealed gap beyond the field median costs points. Method notes · options B and C.

How do I submit my model?

Open an issue in the JevBench repository with a reproducible endpoint or runnable code, the exact model and licence, and whether public JevBench items were used during development. New entrants use the same frozen harness and appear in a new version or a disclosed roster addendum. For private data, see the custom evaluation options.

What a decision costs

Every price here is US dollars per 1,000 decisions — not per 1,000 tokens. One decision is a whole typed request — state, rubric and options — not a single token.

How costs are estimated

Systems with a public tariff (per token or per request) are priced at that tariff times the tokens we measured — 3 rows carry a tariff. Systems without one — open weights, author demos, models we ran ourselves — are priced as if a large inference provider hosted them: the list price of the same weights, or the nearest larger sibling or size class when the exact weights are not listed. We do not use per-minute GPU rental or our own CPU time — providers buy capacity in bulk or own the hardware, and price accordingly. Price × tokens per decision = $ per 1,000 decisions, marked “est.”.

Price rules (v1.5). Only public, bookable list prices that have been in effect for at least 30 days count; a manufacturer's standard, non-promotional launch list price counts from day one, and promotions, subsidies, credits and free tiers never do. The scoring price is never below the market reference price of the system's base model. A system without any eligible price is listed as unpriced — no Cost axis and no score until a price qualifies. A later price change triggers a re-score with a visible note on the row.

  • Cygnet — ~$0.028 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Winnow-12B Q8 — ~$0.028 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Jev 1.13.0 — ~$0.032 est. per 1,000 decisions: operator standard launch list price (interpretation I-1); no exact base-model floor applies
  • Jev-Omni — ~$0.029 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • decider-4b v2 — ~$0.015 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • SemIf — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • spark-s1-4b-v6 — ~$0.021 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • metask-jev-4b — ~$0.026 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • Hopper — ~$0.018 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • Malkuth-4B — ~$0.030 est. per 1,000 decisions: price floor: base-model reference price applied; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • reflex 4B — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • jev-local — ~$0.024 est. per 1,000 decisions: price floor: base-model reference price applied
  • djev (Maisa, diffusion-gemma) — ~$0.053 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Raw Qwen3 4B Instruct 2507 direct logits — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • jqv — ~$0.042 est. per 1,000 decisions: documented hosted-model estimate
  • JevK5 v0.2.0 — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • Qwen3.5-9B Jev-like data-mix v2 — ~$0.065 est. per 1,000 decisions: documented hosted-model estimate
  • Standard One 8B — ~$0.078 est. per 1,000 decisions: documented hosted-model estimate
  • NInfer Qwen3.8-Flash-Next mixed — ~$0.082 est. per 1,000 decisions: documented hosted-model estimate
  • swanOne — ~$0.085 est. per 1,000 decisions: price floor: base-model reference price applied
  • Raw Qwen3 8B direct logits — ~$0.065 est. per 1,000 decisions: documented hosted-model estimate
  • decider-2b — ~$0.015 est. per 1,000 decisions: documented hosted-model estimate
  • system-one — ~$0.068 est. per 1,000 decisions: documented hosted-model estimate
  • system-one-open — ~$0.011 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • GPT-6 Luna (low reasoning effort) — ~$0.108 est. per 1,000 decisions: operator standard launch list price (interpretation I-1); no exact base-model floor applies
  • GPT-6 Luna (default medium reasoning effort) — ~$0.114 est. per 1,000 decisions: operator standard launch list price (interpretation I-1); no exact base-model floor applies
  • JevOne — ~$0.101 est. per 1,000 decisions: documented hosted-model estimate
  • kev 4B — ~$0.014 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • kev 8B — ~$0.097 est. per 1,000 decisions: price floor: base-model reference price applied
  • open-alternative-jev — ~$0.017 est. per 1,000 decisions: ESTIMATE (proxy tokens, I-2): exact base-model market reference; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • Bespoke Nimble 9B — ~$0.128 est. per 1,000 decisions: documented hosted-model estimate
  • Malkuth-2B — ~$0.014 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • openjev-sglang — ~$0.140 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
  • decider-35b-a3b — ~$0.154 est. per 1,000 decisions: price floor: base-model reference price applied
  • local-jev Qwen3.5-4B — ~$0.023 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • Open-Jev 9B — ~$0.170 est. per 1,000 decisions: documented hosted-model estimate
  • Decision 2B — ~$0.013 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • typecastlm — ~$0.016 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • JEV Qwen3.5-9B Base NVFP4 — ~$0.056 est. per 1,000 decisions: ESTIMATE (proxy tokens, I-2): exact base-model market reference
  • NInfer Qwen3.8-27B NVFP4 — ~$0.231 est. per 1,000 decisions: price floor: base-model reference price applied
  • NInfer Qwen3.8-27B NVFP4 (T=1.5) — ~$0.231 est. per 1,000 decisions: price floor: base-model reference price applied
  • Instinct — ~$0.230 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
  • OpenJev (thinking, BF16) — ~$0.241 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • djev (thinking) — ~$0.249 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • LitJev — ~$0.244 est. per 1,000 decisions: price floor: base-model reference price applied
  • Raw Phi-4 mini direct logits — ~$0.036 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • OpenSourceJev — ~$0.011 est. per 1,000 decisions: price floor: base-model reference price applied; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • reflex-27b — ~$0.297 est. per 1,000 decisions: price floor: base-model reference price applied
  • Open-Jev 2B — ~$0.170 est. per 1,000 decisions: documented hosted-model estimate
  • GLiNER2 large — ~$0.0056 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • Qwen3-Reranker-4B — ~$0.052 est. per 1,000 decisions: ESTIMATE: hosted exact-model reference (deepinfra:Qwen/Qwen3-Reranker-4B); no exact base-model floor applies
  • DeepSeek V4.1 Flash — ~$0.498 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
  • SimpleJev — ~$0.0098 est. per 1,000 decisions: documented hosted-model estimate
  • SimpleJev Qwen3.8-27B — ~$0.687 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
  • decision-machine-1 — ~$0.029 est. per 1,000 decisions: operator standard launch list price (interpretation I-1); no exact base-model floor applies
  • GLiNER2.5 multi — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • GLiNER2 — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • JevAct — ~$0.011 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • CLM-8B — ~$0.045 est. per 1,000 decisions: price floor: base-model reference price applied
  • kev 0.6B — ~$0.0046 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Raw Qwen3 0.6B direct logits — ~$0.0056 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • GLiNER2.5 small — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • Raw Qwen3 1.7B direct logits — ~$0.011 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Mirror — ~$0.0023 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • ZeroEntropy zerank-2 — ~$0.052 est. per 1,000 decisions: ESTIMATE: size-class proxy (deepinfra:Qwen/Qwen3-Reranker-4B); no exact base-model floor applies
  • jeff — ~$0.0043 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • smalljev semantic-v9 — ~$0.020 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • OpenDecision — ~$0.0050 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • BAAI bge-reranker-v2-m3 — ~$0.023 est. per 1,000 decisions: ESTIMATE: base-model market reference (deepinfra:BAAI/bge-m3)
  • Certo v1 — ~$0.0013 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Decision Fast — ~$0.0046 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Alibaba GTE Reranker ModernBERT-base — ~$0.011 est. per 1,000 decisions: ESTIMATE: size-class proxy (deepinfra:thenlper/gte-base); no exact base-model floor applies
  • kev 0.5B — ~$0.0046 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Laya — ~$0.0032 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • lev-350m — ~$0.0046 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Qwen3.5-0.8B Decision Model — ~$0.0048 est. per 1,000 decisions: documented hosted-model estimate
  • Mixedbread mxbai-rerank-base-v2 — ~$0.021 est. per 1,000 decisions: ESTIMATE: size-class proxy (deepinfra:Qwen/Qwen3-Reranker-0.6B); no exact base-model floor applies
  • Needle 3 — ~$0.019 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • Needle 3, options as tools — ~$0.019 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • open-jev-deberta-v3-large — ~$0.0056 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • Open Jev JSON Canvas — ~$0.049 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • openJev Verdict — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • openJev Verdict 1.4 — ~$0.0028 est. per 1,000 decisions: ESTIMATE: v1.5 measured-input proxy; no exact base-model floor applies
  • Qwen3.8 27B — ~$2.18 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)
  • verdict-small — ~$0.0009 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • Von — ~$0.0038 est. per 1,000 decisions: documented hosted-model estimate; no exact base-model floor applies
  • classifier.dev — ~$0.023 est. per 1,000 decisions: ESTIMATE (proxy tokens, I-2): operator list price: higher of 19 Sep plan cost and 26 Sep usage tariff USD 0.042/M input (rule 1.2); no exact base-model floor applies
  • JevK5 v0.3 — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • Plumb-4B — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • Decision 4B v1.2 — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • Imajev-4B — ~$0.017 est. per 1,000 decisions: ESTIMATE (I-2): measured input tokens; zero generated output tokens for signed logits readout. M2 floor uses the 25 Sep 2026 DeepInfra Qwen3.5-4B snapshot rates (USD 0.03/M input, USD 0.15/M output); the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen3.5-9B.
  • Decision 4B v1.1 — ~$0.017 est. per 1,000 decisions: documented hosted-model estimate; reference deepinfra:Qwen/Qwen3.5-4B frozen at the 25 Sep cut-off; the snapshot records deprecation on 11 Jun 2026 and replacement by Qwen/Qwen3.5-9B; the frozen snapshot rates remain in use (I-3)
  • Surogate Rune 26B-A4B v3 — ~$0.050 est. per 1,000 decisions: ESTIMATE: M2 base-model market reference from the frozen 25 Sep 2026 OpenRouter snapshot applied to measured tokens.
  • AutoJev-27B (denis-pplx, Qwen3.8-27B) — ~$0.226 est. per 1,000 decisions: documented hosted-model estimate
  • AutoJev-27B (RTX PRO 6000) — ~$0.226 est. per 1,000 decisions: ESTIMATE: M2 base-model market reference from the frozen 25 Sep 2026 OpenRouter snapshot applied to measured tokens.
  • Eikos-27B — ~$0.238 est. per 1,000 decisions: documented hosted-model estimate
  • SimpleJev Qwen3.6-35B-A3B — ~$0.145 est. per 1,000 decisions: ESTIMATE: exact base-model market reference (operator tariff excluded by 30-day rule)

Roster addendum: newcomers scored on the same frozen protocol (9)

All 9 A1/A2 systems below completed all 1,624 decisions and now have official ranks in A, B, and C; the interactive score presets also include them. Their scores, individual 95% intervals, and the frozen v1.5.0 G_med are unchanged. No new paired-bootstrap comparisons were computed for addendum systems. Existing tie markers are retained only for base-system pairs that remain adjacent; no tie or separation is inferred for the other pairs.

SystemRank (A)A score · 95% CIRank (B)B score · 95% CIRank (C)C score · 95% CI
JevK5 v0.3v1.5 roster addendum A1#471.9 69.4–72.9#568.1 64.9–69.8#863.2 51.3–72.1
Plumb-4Bv1.5 roster addendum A1#571.6 69.2–72.7#667.7 64.7–69.6#962.0 50.7–70.9
Decision 4B v1.2v1.5 roster addendum A1#870.8 68.5–72.0#966.6 63.7–68.4#1256.7 47.6–65.1
Imajev-4Bv1.5 roster addendum A2#970.4 67.8–71.6#1166.2 63.1–68.1#1355.9 47.1–64.5
Decision 4B v1.1v1.5 roster addendum A1#1070.4 66.9–71.6#1266.1 62.1–67.9#1455.1 46.4–63.5
Surogate Rune 26B-A4B v3v1.5 roster addendum A2#1666.5 65.4–67.0#1066.6 65.1–67.5#666.5 65.4–67.0
AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1#4919.5 19.3–19.6#4620.4 20.1–20.6#4519.5 19.3–19.6
AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2#5019.5 19.3–19.6#4720.4 20.1–20.6#4619.5 19.3–19.6
Eikos-27Bv1.5 roster addendum A1#5218.5 18.3–18.6#4819.5 19.2–19.7#4818.5 18.3–18.6

Ranks follow option scores; score intervals are per system, not pairwise rank comparisons. Every slider preset sorts these same ranked systems using its selected weights.

Not ranked: partial, unpriced and unmeasured systems

These systems are part of the 103-system v1.5 roster but have no rank. Their numbers are never shown as zero or free.

Partial runs (1)

  • SimpleJev Qwen3.6-35B-A3BAPIpartial run: Partial run: 677 of 1,624 decisions answered; the missing ones count wrong and the row is not ranked.

Incomplete or not measured in v1.5 (3)

  • Jobe Qwen3.5-4B (frozen): not measured.
  • mica-v01-4bv1.5 roster addendum A1: The frozen refusal policy stopped the run after 1,088 of 1,624 rows: 27 documented refusals were mapped to HTTP 422, then three consecutive passthrough HTTP 400 refusals triggered exit 6. The remaining 536 rows have no scores, so this system is not eligible for an official rank. (1,088/1,624 rows; 536 missing).
  • OpenJev (DiffusionGemma 26B-A4B NVFP4, razorback16): not measured.

Incomplete and unmeasured systems receive no official rank. Existing results from earlier benchmark versions remain on their frozen version pages.

Method notes: what changed in v1.5

The four axes

Intelligence · 25%
How often answers are right above chance: each type is normalized against its task-specific random baseline (chance = 0, perfect = 100; below-chance tiers can be negative). Choice, Noul and Score count one third each. Easy, Standard, Judge and Hard items count 10%, 20%, 30% and 40%; the open and sealed sets count equally.
Calibration · 25%
How closely stated probabilities match what happens. It uses ECE and TVD for Choice, ECE and Brier for Noul, and normalized RPS plus top-level ECE for Score. The three types count equally; open and sealed items are pooled.
Speed · 25%
Serial response latency on open Standard and Judge items. The p50 and p95 each get a log score: 100 − 20 × log₁₀(seconds ÷ 0.1), then are averaged. Self-hosted and demo endpoints get the published ×2 plus 0.15-second adjustment.
Cost · 25%
Estimated or billed US dollars per 1,000 decisions, using pooled token use across 1,624 decisions and the documented price rules. The log score is 100 − 30 × log₁₀(cost ÷ $0.001). The price reference is $0.001 per 1,000 decisions.

The official score is the weighted harmonic mean of the four axes. Option A gives each axis 25%; option B (40/20/20/20) is a secondary view, while option C keeps equal axes and sets the Intelligence gate at 60. In A and B, Intelligence, Speed and Cost each have a quadratic gate below 50. To limit benchmaxxing on the public items, the open-minus-sealed Intelligence gap may be up to 8 points above the field median (G_med) before a penalty applies. Each further point lowers the multiplier on unpenalized Intelligence by one percentage point.

Frozen method METHOD-v1.5, SHA-256 c25d3d8b8512…; pricing addendum v1.5-M2, SHA-256 2fc44459ef80….

The method owner chose equal axis weights and equal weights for Choice, Noul and Score after reviewing the What-If Lab, preserving continuity with v1.4 and treating the three decision types equally. Disclosed headline amendment: equal-axis, equal-type A, SHA-256 752ddccc4e19…. B remains a secondary view.

  • 1,624 decisions per system: 904 open (601 published) and 720 sealed, drawn fresh from a private pool with the same tier mix as the open set. Sealed counts for 50% of Intelligence: base = 0.5 × I_open + 0.5 × I_sealed.
  • Three request types are scored natively and chance-corrected per item: Choice, Noul and Score each receive one third. Tier weights easy / standard / judge / hard = 10 / 20 / 30 / 40. A type a system does not support is excluded, never scored zero; only full-coverage systems are ranked.
  • Overfit penalty relative to the field: excess = gap − G_med, penalty = max(0, 1 − max(0, excess − 8) / 100). G_med for this batch is 5.2 CC points.
  • Calibration is typed (Choice ECE/TVD, Noul ECE with Brier, Score normalised RPS and top-level ECE), pooled over open and sealed. Speed and Cost formulas are unchanged from v1.4; self-hosted and demo endpoints carry the ×2 + 0.15 s adjustment. A manufacturer's standard, non-promotional launch list price counts from day one, but a newer price cut younger than 30 days does not. Rows without token counts use the measured proxy-token basis. A system without any eligible public, bookable price is listed as unpriced.
  • The frozen 25 Sep DeepInfra snapshot records Qwen3.5-4B as deprecated on 11 Jun 2026 and replaced by Qwen3.5-9B. Its frozen snapshot rates remain the v1.5 M2 reference; price basis tooltips and the correction note disclose this. Pricing disclosure correction SHA-256: 1b660648bd49….
  • Composite: weighted harmonic mean with the Intelligence, Speed and Cost gates below 50 (Intelligence below 60 in option C). The official headline A uses equal 25 / 25 / 25 / 25 axis weights and Intelligence floor 50. B remains the secondary 40 / 20 / 20 / 20 view; C keeps equal axes and Intelligence floor 60. Ties come from the paired bootstrap.
  • The nine full-coverage systems marked with a v1.5 roster addendum label are officially ranked in A, B, C, and every score preset. Their scores, individual intervals, method, pricing rules and frozen G_med are unchanged. No new paired-bootstrap comparison is inferred for addendum pairs.
  • Before every release we review the leaderboard for anomalies and close loopholes with general, documented rules. The page and Git repository provide transparent data and method details; Benchmark Heaven owns its rules.

Data file SHA-256 6f2fa547454b1108fad701ef302f48450742562393d532d45eccd048f736a9e2 · scorer output SHA-256 452885de2a84cd5b9ed393d541fd8d6c6a540f9d7f762f9d2df9349383f26173 · run kind official.

Limits
  • 1,624 decisions per system (904 open, 720 sealed) is a measurement, not a census, and it is English-only.
  • The weights are a choice. Option A weights the four axes equally and uses a harmonic mean, so the weakest axis dominates; options B and C are published alternatives and the weight sliders re-score the same axes for exploration — only the official option gives the official score and rank. If a wrong decision costs you more than a slow or expensive one, read the Intelligence column and the per-type competence rather than the score alone.
  • The latency adjustment (×2, +0.15 s on our own servers and demo endpoints) is an assumption, not a measurement. We ran the self-hosted and demo endpoints one request at a time (parallelism 1, no other load), so their latency is likely better than the same model on a busy production server. Serving under load trades per-user speed for throughput. The +0.15 s stands for infrastructure our self-hosted tests lacked: authentication, load balancing, logging, billing and an API gateway. Both numbers are assumptions; raw p50/p95 latencies are in the table and the repo.
  • Held-out decisions are sent to the evaluated services to get predictions. Not public is not the same as not seen.
  • Latency is one origin at one time of day; hosted endpoints, public demos and our own pods are different kinds of latency. Public demo endpoints are shared with everyone else using them.
  • Estimated costs describe what a large inference provider would charge for a model of that size, not what the author pays; a system on a tariff pays its tariff.
Credit

Harness, public tasks and every scoring rule: github.com/fstandhartinger/jevbench (MIT). Each project links its author's repository or vendor page.

Authors: if we tested the wrong configuration, tell us and we will rerun it. New entrants become a new version or a disclosed roster addendum rather than silently changing this one.

3D view: three.js r128 (MIT).

Loading capability views…

Loading context-length views…

Previous release: JevBench v1.5.0 (frozen results).

Historical v1.3.0 board: weightings, per-task grid, topic radars and held-out diagnostics

Open this section to load the earlier public-only board and diagnostics.