Hosting = where inference runs; company = where the provider or lab is registered.
EU hosting means the route's inference runs inside the EU: an EU region, AWS Bedrock's EU cross-region (geo) profiles, Azure's Europe Data Zone, or a provider whose entire public fleet is documented as EU-hosted, each checked per model against the provider's documentation. Global deployments do not count, and neither does an EU billing region, an EU company or an EU control plane on its own. One disclosed company-policy exception stays in, marked “EU equivalent”.
Evidence requirements (benchmark evidence, priced provider, measured task tokens) sit above the table they apply to.
Applies to price views & model offers; benchmark evidence stays unfiltered.
JevBench is Benchmark Heaven's own benchmark for Jev-class decision models: state and a bounded rubric in, a typed answer out.
Frozen release · 1,624 decisions per system (904 open + 720 sealed; sealed decisions are half of Intelligence) · 106 ranked of 112 roster systems · only system-level sealed aggregates are published · aggregate results JSON sha256 0cf210b76bf8…
Making decisions from images? Explore Image JevBench v0.1.5 and compare its systems.
Share this version · View live board · Previous release: JevBench v1.5.3
JevBench v1.5.4 · headline
Capability ranking of Jev-class systems
Capability Score averages Intelligence and Calibration. Jev 1.13.0 leads the Jev-class systems with 80.0.
Jev-class means at most 2× the cost and median latency of Jev. How we choose ↘
Wide coloured bar = Capability Score (0–100). Thin bars: cost above, median latency below; shared log ratio scale 0.25× … 64× Jev, ticks at 1× and 2× (cap). Shorter is cheaper or faster. * = est. (estimated cost). # counts ranked Jev-class systems; “–” marks unranked or outside systems. Tap ⓘ or hover a row or thin bar for details.
Cost and latency are shown separately because they are nearly independent across systems (Spearman ρ = 0.07, n = 106).
Sorted by Capability Score, not numbered. Each row says which limit it misses, measured against Jev 1.13.0 ($0.032 per 1,000 decisions, median 0.62 s).
Jev-class = cost per decision at most 2× Jev 1.13.0's (≤ $0.065 per 1,000 decisions) and median latency at most 2× Jev 1.13.0's (≤ 1.23 s), the adjusted p50 — the same median the speed chart plots, not the four-axis Speed score. 60 of 106 systems qualify; the other 46, including the general-purpose LLMs, are listed below the divider in the ranking. The charts below show speed and cost beside Capability Score; the official JevBench Score weighs all four axes.
Jev-class systems are shown by default. Bubble size follows the official JevBench Score. The five most capable Jev-class systems are labelled.
Upper right is best: more capable and cheaper. Cost is USD per 1,000 decisions on a log scale. The dashed line is 2× Jev’s cost.
Upper right is best: more capable and faster. Speed here is the median-latency speed — the same adjusted median (p50) latency the Jev-class limit uses — a log scale, so each 20 points is 10× faster (median under the numbers). The dashed line is 2× Jev’s median latency.
What-If: the weight sliders below re-score every system under other axis weights — only the equal 25/25/25/25 weights give the official option-A ranking. The 3D view of capability, cost and speed loads further down.
JevBench v1.5.4
Official· four axes 0–100, equal-weight harmonic mean · Method notes ↓
Cygnet and Winnow-12B Q8 are joint leaders (statistical tie).
Whiskers are 95% bootstrap intervals. 69 of 77 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other 28 adjacent pairs, so no tie classification is inferred.
Greener = stronger within its column.
106 of 106 systems, sorted by official rank, #1 first.
Weights are relative: each axis counts in proportion to its slider. The score stays a weighted harmonic mean with the low-axis gates; the gates still apply when an axis sits at 0. Only equal weights give the official JevBench Score and rank.
Score = 4 / (1/I + 1/C + 1/S + 1/K) (each 0–100; × (axis / 50)² for Intelligence, Speed or Cost below 50)
system-one-openEligibility rule: services that run on Jev itself may be measured and shown as honorable mentions, but are not competitors ranked against Jev and do not enter the field median gap (G_med) or tie markers. classifier.dev (TypeSafe) runs on Jev, so it stays unranked under this rule.
Pick any two. Radars for the score axes, capability by subject topic and use cases, chance-corrected competence per request type on the open and sealed sets, and competence per tier on each set. Further out is better on every spoke; the link keeps the pair.
system-one-open · Score 73.7 (#1)Not drawn: Other — none of the above: general reasoning, arithmetic or date puzzles, everyday assistant requests without a business workflow. 159 items (76 open / 83 sealed) — not a use case of its own, so it is counted but not drawn.
Low n (under 15 items, not plotted): Scientific discovery (1), Semantic code linting (4), Feature extraction for predictive modeling (7), Recruiting (3), Moderation and trust and safety (12), Advertising (2), Gaming (0), Demand forecasting (0).
Chance-corrected competence (0 = chance, 100 = perfect) over the category's items, open and sealed pooled, all tiers: per request type the method's competence cell (Choice and Noul accuracy above chance, Score 1 - error / chance error), then averaged over types weighted by item count. Not part of the JevBench Score; compare systems within a category, not categories with each other. Values can be negative (below chance).
Every one of the 1,624 decisions was labelled with one subject topic (the v1.2 topic list, unchanged) and one or two TypeSafe use-case categories (docs.typesafe.ai/concepts/use-case-map, plus Other) by Winnow-12B Q8 (the model behind System1 Models s1-pro) on our own GPU pod; 1,118 of the 3,248 calls were first made on s1-pro itself and agree 98.2 % with the pod run. Sealed items stayed on our own infrastructure. About 5 % of public items were checked by hand (Claude Opus 5.5); the item-group rules below fix the systematic misses found.
| Spoke | A: Jev 1.13.0 | B: Cygnet |
|---|---|---|
| The four score axes | ||
| Intelligence | 72.0 | 71.1 |
| Calibration | 88.0 | 87.0 |
| Speed | 83.8 | 91.0 |
| Cost | 54.7 | 56.4 |
| Capability by subject topic | ||
| Math & numbers | 40.5 | 31.4 |
| Coding & software | 80.3 | 87.8 |
| Rules, policy & law | 73.7 | 76.2 |
| Finance & commerce | 83.4 | 78.5 |
| Support & operations | 91.5 | 88.2 |
| Everyday language | 96.1 | 95.6 |
| Safety & security | 67.7 | 74.4 |
| Use cases (TypeSafe categories) | ||
| Search & retrieval | 95.9 | 94.7 |
| Model routing | 96.5 | 99.1 |
| LLM guardrails | 67.0 | 72.3 |
| Lead generation | 88.1 | 82.2 |
| Customer support | 91.9 | 88.9 |
| Insurance claims | 67.5 | 71.7 |
| Financial crime | 58.3 | 71.1 |
| Legal & compliance | 74.0 | 75.3 |
| E-commerce | 89.8 | 91.6 |
| Risk assessment | 80.1 | 73.1 |
| Knowledge graphs | 100.0 | 73.2 |
| Competence per request type, open / sealed | ||
| Choice · open | 85.7 | 82.9 |
| Choice · sealed | 87.6 | 80.3 |
| Noul · open | 47.8 | 57.9 |
| Noul · sealed | 48.6 | 55.7 |
| Score · open | 81.2 | 76.6 |
| Score · sealed | 81.1 | 73.2 |
| Competence per tier — open set | ||
| Easy | 90.0 | 89.7 |
| Standard | 80.9 | 82.4 |
| Judge | 83.5 | 82.2 |
| Hard | 53.8 | 58.0 |
| Competence per tier — sealed set | ||
| Easy | 91.2 | 88.6 |
| Standard | 79.0 | 73.5 |
| Judge | 73.7 | 75.4 |
| Hard | 72.9 | 65.3 |
Every measured system, under the four score axes. Intelligence is 50% open (904 decisions) and 50% sealed (720); Gap = I_open − I_sealed, and the penalty applies only above the field median gap (G_med 5.2) plus 8. The per-type competence, latency and price of the same systems are in Types & cost. On a phone the name column stays put while the table scrolls sideways.
| #A | System | Score | Intel. | Calib. | Speed | Cost | Gap | Penalty |
|---|---|---|---|---|---|---|---|---|
| 1 | CygnetdetailsBase model: google/gemma-4-12B-itsource | 73.7 | 71.1 | 87.0 | 91.0 | 56.4 | +2.7 | ×1.000 |
| 2 | Winnow-12B Q8detailsBase model: google/gemma-4-12B-itsource | 73.2 | 74.4 | 84.1 | 86.1 | 56.6 | -3.9 | ×1.000 |
| 3 | Jev 1.13.0APIdetailsBase model: undisclosed | 72.1 | 72.0 | 88.0 | 83.8 | 54.7 | -0.9 | ×1.000 |
| 4 | JevK5 v0.3v1.5 roster addendum A1Base model: Qwen/Qwen3.5-4Bsource | 71.9 | 56.3 | 88.3 | 93.6 | 63.1 | +10.7 | ×1.000 |
| 5 | Plumb-4Bv1.5 roster addendum A1Base model: alibiserikbay/JevK5source | 71.6 | 55.8 | 87.4 | 93.5 | 63.1 | +10.1 | ×1.000 |
| 6 | Jev-OmnidetailsBase model: google/gemma-4-12B-itsource | 71.5 | 70.5 | 82.6 | 84.7 | 56.1 | -0.0 | ×1.000 |
| 7 | decider-4b v2detailsBase model: Qwen/Qwen3.5-4B-Basesource | 71.3 | 55.8 | 85.6 | 90.9 | 64.5 | +11.9 | ×1.000 |
| 8 | Decision 4B v1.2v1.5 roster addendum A1Base model: undisclosed | 70.8 | 53.7 | 88.6 | 93.5 | 63.1 | +5.2 | ×1.000 |
| 9 | Imajev-4Bv1.5 roster addendum A2Base model: Qwen/Qwen3.5-4Bsource | 70.4 | 53.5 | 88.1 | 91.1 | 63.3 | +5.5 | ×1.000 |
| 10 | Decision 4B v1.1v1.5 roster addendum A1Base model: undisclosed | 70.4 | 53.1 | 87.3 | 93.5 | 63.1 | +3.1 | ×1.000 |
| 11 | Manchego v2.1v1.5 roster addendum A5detailsBase model: Qwen/Qwen3.5-4Bsource | 68.8 | 51.2 | 84.9 | 88.6 | 64.3 | +6.4 | ×1.000 |
| 12 | SemIfdetailsBase model: Qwen/Qwen3.5-4Bsource | 68.7 | 51.3 | 84.0 | 90.9 | 63.1 | +5.4 | ×1.000 |
| 13 | spark-s1-4b-v6detailsBase model: Qwen3.5-4Bsource | 68.2 | 62.1 | 69.7 | 85.5 | 60.4 | +0.7 | ×1.000 |
| 14 | metask-jev-4bdetailsBase model: Qwen3.5-4Bsource | 67.5 | 53.5 | 82.7 | 89.5 | 57.7 | +6.5 | ×1.000 |
| 15 | HopperdetailsBase model: Qwen/Qwen3.5-4Bsource | 67.5 | 49.9 | 87.9 | 87.2 | 62.3 | -2.3 | ×1.000 |
| 16 | Malkuth-4BdetailsBase model: Qwen/Qwen3.5-4B-Basesource | 66.8 | 54.5 | 83.2 | 86.2 | 55.8 | +7.8 | ×1.000 |
| 17 | Surogate Rune 26B-A4B v3v1.5 roster addendum A2Base model: google/gemma-4-26B-A4B-itsource | 66.5 | 69.7 | 88.3 | 86.0 | 49.0 | -1.6 | ×1.000 |
| 18 | reflex 4BdetailsBase model: Qwen/Qwen3.5-4Bsource | 65.2 | 51.5 | 86.8 | 68.8 | 63.1 | +7.4 | ×1.000 |
| 19 | jev-localdetailsBase model: Qwen/Qwen3.5-9Bsource | 65.2 | 56.3 | 77.8 | 73.0 | 58.9 | +3.8 | ×1.000 |
| 20 | djev (Maisa, diffusion-gemma)detailsBase model: google/diffusiongemma-26B-A4B-itsource | 64.2 | 72.3 | 80.4 | 91.0 | 48.2 | +0.8 | ×1.000 |
| 21 | Raw Qwen3 4B Instruct 2507 direct logitsdetailsBase model: undisclosedsource | 62.1 | 54.1 | 52.9 | 89.2 | 63.4 | +6.7 | ×1.000 |
| 22 | jqvdetailsBase model: Qwen3-32Bsource | 60.8 | 49.1 | 86.8 | 83.4 | 51.2 | +4.5 | ×1.000 |
| 23 | JevK5 v0.2.0detailsBase model: Qwen3.5-4Bsource | 58.1 | 46.7 | 84.9 | 90.9 | 63.1 | +6.6 | ×1.000 |
| 24 | Qwen3.5-9B Jev-like data-mix v2detailsBase model: Qwen/Qwen3.5-9Bsource | 53.0 | 60.4 | 80.9 | 82.0 | 45.7 | +4.3 | ×1.000 |
| 25 | Standard One 8BdetailsBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16source | 47.8 | 59.6 | 83.2 | 92.5 | 43.3 | +0.7 | ×1.000 |
| 26 | NInfer Qwen3.8-Flash-Next mixeddetailsBase model: Qwen3.8-Flash-Nextsource | 47.5 | 67.2 | 88.5 | 88.6 | 42.5 | +6.1 | ×1.000 |
| 27 | Instinct Dual 4Bv1.5 roster addendum A4APIdetailsBase model: Qwen3.5-4Bsource · Operator-reported; weights not publicly verifiable. | 47.0 | 43.1 | 88.3 | 82.0 | 60.2 | +6.3 | ×1.000 |
| 28 | swanOnedetailsBase model: Mia-AiLab/Qwen3.8-Flash-Next-NVFP4source | 46.6 | 71.2 | 87.1 | 84.7 | 42.2 | -1.5 | ×1.000 |
| 29 | Raw Qwen3 8B direct logitsdetailsBase model: Qwen/Qwen3-8B-Basesource | 45.2 | 51.1 | 49.2 | 86.8 | 45.5 | +11.3 | ×1.000 |
| 30 | decider-2bdetailsBase model: Qwen/Qwen3.5-2B-Basesource | 45.1 | 42.3 | 71.5 | 94.4 | 64.9 | +12.9 | ×1.000 |
| 31 | system-onedetailsBase model: Qwen3-8Bsource | 44.1 | 50.5 | 49.4 | 90.9 | 45.0 | +6.1 | ×1.000 |
| 32 | system-one-openAPIdetailsBase model: Gemma 4 E2Bsource | 42.4 | 41.6 | 71.6 | 78.3 | 68.3 | +5.6 | ×1.000 |
| 33 | Autoloops – Gemma 4 31B ITAPIdetailsBase model: undisclosed | 40.5 | 76.7 | 85.8 | 83.9 | 39.6 | +0.4 | ×1.000 |
| 34 | GPT-6 Luna (low reasoning effort)APIdetailsBase model: undisclosed | 40.5 | 95.3 | 94.9 | 73.2 | 39.1 | -1.1 | ×1.000 |
| 35 | GPT-6 Luna (default medium reasoning effort)APIdetailsBase model: undisclosed | 38.8 | 96.2 | 95.6 | 73.2 | 38.3 | +0.2 | ×1.000 |
| 36 | JevOnedetailsBase model: Qwen/Qwen3.6-35B-A3Bsource | 38.2 | 54.1 | 84.9 | 90.4 | 39.8 | -0.5 | ×1.000 |
| 37 | kev 4BdetailsBase model: Qwen/Qwen3-4B-Basesource · Evaluated Qwen3 variant; later releases use a different base. | 38.1 | 39.9 | 67.6 | 85.5 | 65.8 | +13.9 | ×0.993 |
| 38 | kev 8BdetailsBase model: Qwen/Qwen3-8B-Basesource | 34.2 | 48.3 | 71.0 | 84.1 | 40.4 | +12.0 | ×1.000 |
| 39 | open-alternative-jevdetailsBase model: Qwen3.5-4Bsource | 33.6 | 37.4 | 76.9 | 91.3 | 63.3 | +10.5 | ×1.000 |
| 40 | Bespoke Nimble 9BdetailsBase model: Qwen3.5-9Bsource | 31.8 | 63.7 | 77.2 | 83.0 | 36.8 | +2.8 | ×1.000 |
| 41 | Malkuth-2BdetailsBase model: empero-ai/Qwen3.8-2B-Distillsource | 29.9 | 35.5 | 75.2 | 91.8 | 65.6 | +15.6 | ×0.976 |
| 42 | openjev-sglangAPIdetailsBase model: Qwen/Qwen3.6-35B-A3Bsource | 29.0 | 58.6 | 82.9 | 78.1 | 35.6 | +5.4 | ×1.000 |
| 43 | decider-35b-a3bdetailsBase model: Qwen/Qwen3.5-35B-A3B-Basesource | 27.5 | 60.5 | 82.0 | 91.0 | 34.4 | +7.4 | ×1.000 |
| 44 | local-jev Qwen3.5-4BdetailsBase model: Qwen/Qwen3.5-4Bsource | 25.8 | 33.7 | 82.4 | 83.7 | 59.3 | +5.9 | ×1.000 |
| 45 | Nemotron Diffusion 8Bv1.5 roster addendum A3detailsBase model: nvidia/Nemotron-Labs-Diffusion-8Bsource | 25.7 | 33.8 | 76.2 | 93.7 | 55.5 | +11.4 | ×1.000 |
| 46 | Open-Jev 9BdetailsBase model: Qwen/Qwen3.5-9Bsource | 24.4 | 63.8 | 81.5 | 73.6 | 33.1 | -4.1 | ×1.000 |
| 47 | Decision 2BdetailsBase model: openbmb/MiniCPM5-2Bsource | 22.5 | 31.3 | 86.2 | 90.4 | 66.4 | +6.7 | ×1.000 |
| 48 | GPT-5.6 LunaAPIdetailsBase model: undisclosed | 22.4 | 94.3 | 94.7 | 74.1 | 30.7 | -2.3 | ×1.000 |
| 49 | typecastlmdetailsBase model: Qwen3.5-4Bsource | 21.8 | 31.2 | 76.7 | 92.0 | 64.0 | -15.1 | ×1.000 |
| 50 | JEV Qwen3.5-9B Base NVFP4detailsBase model: ig1/Qwen3.5-9B-NVFP4source | 20.1 | 32.2 | 80.9 | 93.5 | 47.6 | +1.3 | ×1.000 |
| 51 | Gemini 3.1 Flash-LiteAPIdetailsBase model: undisclosed | 19.6 | 77.6 | 74.7 | 80.1 | 29.8 | -1.9 | ×1.000 |
| 52 | AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1Base model: Qwen/Qwen3.8-27Bsource | 19.5 | 72.8 | 87.7 | 87.6 | 29.4 | -1.7 | ×1.000 |
| 53 | AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2Base model: Qwen/Qwen3.8-27Bsource | 19.5 | 72.8 | 86.7 | 87.1 | 29.4 | -1.5 | ×1.000 |
| 54 | NInfer Qwen3.8-27B NVFP4detailsBase model: Qwen3.8-27Bsource | 18.7 | 65.5 | 85.9 | 89.9 | 29.1 | +3.0 | ×1.000 |
| 55 | Eikos-27Bv1.5 roster addendum A1Base model: Qwen/Qwen3.8-27Bsource | 18.5 | 75.1 | 86.3 | 87.6 | 28.7 | -1.9 | ×1.000 |
| 56 | NInfer Qwen3.8-27B NVFP4 (T=1.5)detailsBase model: Qwen3.8-27Bsource | 18.5 | 61.1 | 86.5 | 89.9 | 29.1 | +2.7 | ×1.000 |
| 57 | InstinctAPIdetailsBase model: Qwen3.8-27Bsource · Operator-reported; weights not publicly verifiable. | 18.3 | 62.7 | 85.0 | 81.7 | 29.2 | -1.7 | ×1.000 |
| 58 | OpenJev (thinking, BF16)detailsBase model: undisclosedsource | 17.9 | 84.2 | 83.1 | 73.9 | 28.5 | -3.4 | ×1.000 |
| 59 | djev (thinking)detailsBase model: google/diffusiongemma-26B-A4B-itsource | 17.4 | 77.3 | 95.7 | 72.3 | 28.1 | -1.6 | ×1.000 |
| 60 | LitJevdetailsBase model: Qwen/Qwen3.8-27Bsource | 16.3 | 58.3 | 84.5 | 68.2 | 28.4 | -0.5 | ×1.000 |
| 61 | Bev / Bonsai 27Bv1.5 roster addendum A4detailsBase model: Qwen/Qwen3.8-27Bsource | 15.8 | 53.1 | 77.7 | 72.7 | 28.2 | -8.8 | ×1.000 |
| 62 | Raw Phi-4 mini direct logitsdetailsBase model: undisclosedsource | 15.2 | 27.6 | 71.4 | 89.2 | 53.2 | +18.3 | ×0.949 |
| 63 | OpenSourceJevdetailsBase model: Qwen3.5-4Bsource | 13.2 | 25.8 | 76.0 | 72.5 | 69.1 | +4.2 | ×1.000 |
| 64 | reflex-27bdetailsBase model: Qwen3.8-27Bsource | 13.2 | 62.8 | 85.9 | 69.2 | 25.8 | +0.6 | ×1.000 |
| 65 | Open-Jev 2BdetailsBase model: Qwen/Qwen3.5-2Bsource | 9.1 | 33.6 | 73.6 | 75.8 | 33.1 | +10.4 | ×1.000 |
| 66 | GLiNER2 largedetailsBase model: microsoft/deberta-v3-largesource | 8.4 | 22.5 | 42.4 | 64.8 | 77.6 | +7.1 | ×1.000 |
| 67 | Qwen3-Reranker-4BdetailsBase model: Qwen/Qwen3-4B-Basesource | 7.1 | 21.0 | 76.3 | 79.6 | 48.4 | +17.0 | ×0.962 |
| 68 | DeepSeek V4.1 FlashAPIdetailsBase model: undisclosed | 6.6 | 93.7 | 96.9 | 69.4 | 19.1 | -3.7 | ×1.000 |
| 69 | SimpleJevdetailsBase model: Qwen/Qwen3.5-0.8Bsource | 4.0 | 16.7 | 46.7 | 59.1 | 70.3 | -6.8 | ×1.000 |
| 70 | SimpleJev Qwen3.8-27BAPIdetailsBase model: Qwen/Qwen3.8-27Bsource | 3.4 | 72.8 | 87.2 | 74.9 | 14.9 | -1.7 | ×1.000 |
| 71 | decision-machine-1APIdetailsBase model: undisclosed | 3.2 | 14.8 | 81.2 | 92.8 | 56.3 | +22.4 | ×0.908 |
| 72 | GLiNER2.5 multidetailsBase model: mDeBERTa-v3-basesource | 2.7 | 14.0 | 58.0 | 66.6 | 86.6 | +3.6 | ×1.000 |
| 73 | Bosun v3.1 0.6Bv1.5 roster addendum A4detailsBase model: Qwen/Qwen3-0.6Bsource | 2.5 | 13.6 | 65.2 | 61.8 | 77.5 | +17.8 | ×0.954 |
| 74 | GLiNER2detailsBase model: DeBERTa-v3-basesource | 2.3 | 13.5 | 35.6 | 70.3 | 86.6 | +7.5 | ×1.000 |
| 75 | Deem 0.8B v1v1.5 roster addendum A4detailsBase model: Qwen/Qwen3.5-0.8Bsource | 2.1 | 13.0 | 38.9 | 84.3 | 80.3 | +21.1 | ×0.921 |
| 76 | JevActAPIdetailsBase model: undisclosedsource | 1.5 | 11.2 | 62.5 | 76.4 | 68.6 | +16.3 | ×0.969 |
| 77 | CLM-8BdetailsBase model: Qwen/Qwen3-8Bsource | 1.5 | 11.4 | 48.5 | 93.3 | 50.5 | +11.3 | ×1.000 |
| 78 | kev 0.6BdetailsBase model: Qwen/Qwen3-0.6B-Basesource | 1.3 | 10.3 | 67.7 | 87.2 | 80.1 | +27.0 | ×0.862 |
| 79 | Raw Qwen3 0.6B direct logitsdetailsBase model: Qwen/Qwen3-0.6B-Basesource | 1.1 | 10.6 | 21.5 | 90.4 | 77.6 | -1.1 | ×1.000 |
| 80 | GLiNER2.5 smalldetailsBase model: microsoft/deberta-v3-xsmallsource | 0.9 | 9.2 | 55.9 | 77.4 | 86.6 | +15.5 | ×0.977 |
| 81 | Raw Qwen3 1.7B direct logitsdetailsBase model: Qwen/Qwen3-1.7B-Basesource | 0.9 | 9.7 | 21.6 | 90.2 | 68.5 | +5.2 | ×1.000 |
| 82 | MirrordetailsBase model: undisclosed | 0.2 | 5.6 | 43.2 | 63.6 | 89.3 | +2.3 | ×1.000 |
| 83 | ZeroEntropy zerank-2detailsBase model: Qwen/Qwen3-4Bsource | 0.1 | 4.8 | 81.7 | 80.3 | 48.4 | +10.4 | ×1.000 |
| 84 | jeffdetailsBase model: undisclosedsource | 0.1 | 4.4 | 80.3 | 55.8 | 81.1 | +16.3 | ×0.969 |
| 85 | smalljev semantic-v9detailsBase model: openbmb/MiniCPM5-2B-Basesource | 0.1 | 3.7 | 73.1 | 86.5 | 60.8 | +14.5 | ×0.987 |
| 86 | Laya multilingualv1.5 roster addendum A4detailsBase model: mmBERT-basesource | 0.0 | 2.4 | 43.6 | 73.7 | 82.2 | +0.9 | ×1.000 |
| 87 | OpenDecisiondetailsBase model: MoritzLaurer/ModernBERT-large-zeroshot-v2.0source | 0.0 | 2.1 | 72.5 | 86.6 | 79.1 | +14.7 | ×0.985 |
| 88 | BAAI bge-reranker-v2-m3detailsBase model: BAAI/bge-m3source | 0.0 | 0.0 | 83.3 | 90.5 | 59.3 | +1.7 | ×1.000 |
| 89 | Certo v1detailsBase model: ModernBERT-largesource | 0.0 | 0.0 | 87.5 | 91.4 | 96.9 | +0.8 | ×1.000 |
| 90 | Decision FastdetailsBase model: Qwen/Qwen3-0.6B-Basesource | 0.0 | 0.0 | 76.1 | 91.4 | 80.1 | +19.0 | ×0.942 |
| 91 | Alibaba GTE Reranker ModernBERT-basedetailsBase model: answerdotai/ModernBERT-basesource | 0.0 | 0.0 | 75.2 | 91.5 | 69.3 | +4.5 | ×1.000 |
| 92 | kev 0.5BdetailsBase model: Qwen/Qwen2.5-0.5Bsource | 0.0 | 0.0 | 65.0 | 88.5 | 80.1 | +25.7 | ×0.875 |
| 93 | LayadetailsBase model: ModernBERT-largesource | 0.0 | 0.0 | 73.7 | 73.9 | 84.9 | +12.7 | ×1.000 |
| 94 | lev-350mdetailsBase model: LiquidAI/LFM2.5-350Msource | 0.0 | 0.0 | 77.7 | 93.6 | 80.0 | +23.5 | ×0.896 |
| 95 | Qwen3.5-0.8B Decision ModeldetailsBase model: Qwen/Qwen3.5-0.8B-Basesource | 0.0 | 0.0 | 73.6 | 71.8 | 79.5 | +7.1 | ×1.000 |
| 96 | Mixedbread mxbai-rerank-base-v2detailsBase model: undisclosedsource | 0.0 | 0.0 | 86.6 | 89.3 | 60.3 | +3.0 | ×1.000 |
| 97 | Needle 3detailsBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 34.1 | 61.6 | +10.0 | ×1.000 |
| 98 | Needle 3, options as toolsdetailsBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 41.0 | 61.6 | +14.1 | ×0.991 |
| 99 | open-jev-deberta-v3-largedetailsBase model: microsoft/deberta-v3-largesource | 0.0 | 0.0 | 77.1 | 68.3 | 77.6 | +5.3 | ×1.000 |
| 100 | Open Jev JSON CanvasdetailsBase model: google/diffusiongemma-26B-A4B-itsource | 0.0 | 77.1 | 0.0 | 85.6 | 49.3 | +4.0 | ×1.000 |
| 101 | openJev VerdictdetailsBase model: undisclosedsource | 0.0 | 0.0 | 52.2 | 83.9 | 86.6 | +26.7 | ×0.865 |
| 102 | openJev Verdict 1.4detailsBase model: knowledgator/gliclass-modern-base-v2.0source | 0.0 | 0.0 | 80.3 | 80.6 | 86.6 | +3.5 | ×1.000 |
| 103 | Qwen3.8 27BAPIdetailsBase model: undisclosed | 0.0 | 95.6 | 98.1 | 56.8 | 0.0 | -5.3 | ×1.000 |
| 104 | verdict-smalldetailsBase model: intfloat/multilingual-e5-smallsource | 0.0 | 0.0 | 59.5 | 82.1 | 100.0 | +2.0 | ×1.000 |
| 105 | VondetailsBase model: answerdotai/ModernBERT-largesource | 0.0 | 0.0 | 83.5 | 75.7 | 82.7 | +14.7 | ×0.985 |
| 106 | Laya typed-decisionsv1.5 roster addendum A4detailsBase model: ModernBERT-largesource | 0.0 | 0.0 | 83.3 | 63.4 | 82.5 | +8.5 | ×1.000 |
| – | classifier.devAPIhonorable mentiondetailsBase model: undisclosed | — | 75.8 | 89.2 | 82.0 | 58.9 | -2.7 | ×1.000 |
| – | SimpleJev Qwen3.6-35B-A3BAPIpartial rundetailsBase model: undisclosedsource | — | 0.0 | 78.5 | 75.2 | 35.1 | +5.6 | ×1.000 |
| – | Decision-4Bv1.5 roster addendum A4not rankeddetailsBase model: Qwen/Qwen3.5-4Bsource | — | — | — | — | — | — | — |
The same systems, by what was asked and what it cost. Per-type columns are chance-corrected competence (CC, 0 = chance) for Choice, Noul and Score, open / sealed; I open and I sealed are the two halves of Intelligence. Latency is adjusted p50 / p95; cost is per 1,000 decisions. On a phone the name column stays put while the table scrolls sideways.
| #A | System | I open | I sealed | Choice o / s | Noul o / s | Score o / s | p50 / p95 | $/1k decisions | Endpoint |
|---|---|---|---|---|---|---|---|---|---|
| 1 | CygnetdetailsBase model: google/gemma-4-12B-itsource | 72.5 | 69.7 | 83 / 80 | 58 / 56 | 77 / 73 | 0.23 s / 0.35 s | $0.028 | GPU pod (RTXPRO6000) |
| 2 | Winnow-12B Q8detailsBase model: google/gemma-4-12B-itsource | 72.5 | 76.4 | 78 / 82 | 59 / 66 | 81 / 81 | 0.34 s / 0.72 s | $0.028 | GPU pod (RTX6000) |
| 3 | Jev 1.13.0APIdetailsBase model: undisclosed | 71.5 | 72.5 | 86 / 88 | 48 / 49 | 81 / 81 | 0.62 s / 0.67 s | $0.032 | hosted API |
| 4 | JevK5 v0.3v1.5 roster addendum A1Base model: Qwen/Qwen3.5-4Bsource | 61.6 | 50.9 | 80 / 74 | 42 / 15 | 62 / 64 | 0.18 s / 0.24 s | $0.017 | GPU pod (H100) |
| 5 | Plumb-4Bv1.5 roster addendum A1Base model: alibiserikbay/JevK5source | 60.9 | 50.8 | 80 / 73 | 43 / 17 | 59 / 63 | 0.18 s / 0.24 s | $0.017 | GPU pod (H100) |
| 6 | Jev-OmnidetailsBase model: google/gemma-4-12B-itsource | 70.5 | 70.5 | 83 / 79 | 55 / 57 | 73 / 76 | 0.38 s / 0.91 s | $0.029 | GPU pod (RTX6000) |
| 7 | decider-4b v2detailsBase model: Qwen/Qwen3.5-4B-Basesource | 61.7 | 49.8 | 80 / 77 | 43 / 19 | 62 / 53 | 0.20 s / 0.40 s | $0.015 | GPU pod (RTX5090) |
| 8 | Decision 4B v1.2v1.5 roster addendum A1Base model: undisclosed | 56.3 | 51.1 | 81 / 74 | 20 / 16 | 68 / 63 | 0.18 s / 0.24 s | $0.017 | GPU pod (H100) |
| 9 | Imajev-4Bv1.5 roster addendum A2Base model: Qwen/Qwen3.5-4Bsource | 56.2 | 50.7 | 80 / 82 | 22 / 8 | 67 / 62 | 0.23 s / 0.33 s | $0.017 | GPU pod (RTX 5090) |
| 10 | Decision 4B v1.1v1.5 roster addendum A1Base model: undisclosed | 54.7 | 51.6 | 80 / 75 | 18 / 16 | 66 / 64 | 0.18 s / 0.24 s | $0.017 | GPU pod (H100) |
| 11 | Manchego v2.1v1.5 roster addendum A5detailsBase model: Qwen/Qwen3.5-4Bsource | 54.4 | 48.0 | 67 / 65 | 34 / 17 | 62 / 62 | 0.32 s / 0.43 s | $0.015 | GPU pod (RTX6000) |
| 12 | SemIfdetailsBase model: Qwen/Qwen3.5-4Bsource | 54.0 | 48.6 | 74 / 71 | 30 / 14 | 58 / 61 | 0.23 s / 0.36 s | $0.017 | GPU pod (RTX5090) |
| 13 | spark-s1-4b-v6detailsBase model: Qwen3.5-4Bsource | 62.5 | 61.8 | 73 / 73 | 50 / 46 | 64 / 66 | 0.41 s / 0.69 s | $0.021 | GPU pod (RTX6000) |
| 14 | metask-jev-4bdetailsBase model: Qwen3.5-4Bsource | 56.7 | 50.2 | 70 / 72 | 43 / 25 | 57 / 53 | 0.29 s / 0.39 s | $0.026 | GPU pod (RTX5090) |
| 15 | HopperdetailsBase model: Qwen/Qwen3.5-4Bsource | 48.7 | 51.0 | 70 / 75 | 10 / 13 | 66 / 65 | 0.39 s / 0.48 s | $0.018 | GPU pod (A6000) |
| 16 | Malkuth-4BdetailsBase model: Qwen/Qwen3.5-4B-Basesource | 58.4 | 50.5 | 71 / 70 | 39 / 23 | 64 / 59 | 0.44 s / 0.55 s | $0.030 | GPU pod (RTX5090) |
| 17 | Surogate Rune 26B-A4B v3v1.5 roster addendum A2Base model: google/gemma-4-26B-A4B-itsource | 69.0 | 70.5 | 82 / 82 | 46 / 50 | 79 / 80 | 0.35 s / 0.72 s | $0.050 | GPU pod (RTX PRO 6000) |
| 18 | reflex 4BdetailsBase model: Qwen/Qwen3.5-4Bsource | 55.2 | 47.8 | 74 / 69 | 33 / 19 | 59 / 55 | 2.9 s / 4.6 s | $0.017 | GPU pod (H100) |
| 19 | jev-localdetailsBase model: Qwen/Qwen3.5-9Bsource | 58.2 | 54.4 | 67 / 65 | 47 / 37 | 61 / 61 | 0.94 s / 5.3 s | $0.024 | GPU pod (H100) |
| 20 | djev (Maisa, diffusion-gemma)detailsBase model: google/diffusiongemma-26B-A4B-itsource | 72.7 | 71.9 | 77 / 73 | 66 / 70 | 75 / 73 | 0.25 s / 0.32 s | $0.053 | GPU pod (H100) |
| 21 | Raw Qwen3 4B Instruct 2507 direct logitsdetailsBase model: undisclosedsource | 57.4 | 50.7 | 65 / 63 | 50 / 38 | 57 / 51 | 0.26 s / 0.46 s | $0.017 | GPU pod (A6000) |
| 22 | jqvdetailsBase model: Qwen3-32Bsource | 51.3 | 46.8 | 73 / 73 | 19 / 10 | 62 / 57 | 0.31 s / 1.5 s | $0.042 | GPU pod (H100) |
| 23 | JevK5 v0.2.0detailsBase model: Qwen3.5-4Bsource | 50.0 | 43.4 | 78 / 72 | 12 / -4 | 60 / 62 | 0.22 s / 0.38 s | $0.017 | GPU pod (RTX5090) |
| 24 | Qwen3.5-9B Jev-like data-mix v2detailsBase model: Qwen/Qwen3.5-9Bsource | 62.6 | 58.3 | 73 / 75 | 42 / 32 | 73 / 68 | 0.58 s / 1.1 s | $0.065 | GPU pod (A6000) |
| 25 | Standard One 8BdetailsBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16source | 60.0 | 59.2 | 74 / 72 | 45 / 43 | 61 / 63 | 0.20 s / 0.28 s | $0.078 | GPU pod (RTX5090) |
| 26 | NInfer Qwen3.8-Flash-Next mixeddetailsBase model: Qwen3.8-Flash-Nextsource | 70.3 | 64.1 | 85 / 79 | 51 / 43 | 74 / 70 | 0.31 s / 0.44 s | $0.082 | GPU pod (RTXPRO6000) |
| 27 | Instinct Dual 4Bv1.5 roster addendum A4APIdetailsBase model: Qwen3.5-4Bsource · Operator-reported; weights not publicly verifiable. | 46.3 | 39.9 | 75 / 74 | 4 / -13 | 59 / 58 | 0.51 s / 1.3 s | $0.021 | author demo endpoint |
| 28 | swanOnedetailsBase model: Mia-AiLab/Qwen3.8-Flash-Next-NVFP4source | 70.5 | 71.9 | 88 / 83 | 46 / 60 | 78 / 73 | 0.57 s / 0.59 s | $0.085 | GPU pod (RTXPRO6000) |
| 29 | Raw Qwen3 8B direct logitsdetailsBase model: Qwen/Qwen3-8B-Basesource | 56.8 | 45.5 | 65 / 57 | 48 / 38 | 57 / 42 | 0.33 s / 0.62 s | $0.065 | GPU pod (A6000) |
| 30 | decider-2bdetailsBase model: Qwen/Qwen3.5-2B-Basesource | 48.8 | 35.9 | 52 / 53 | 35 / 6 | 59 / 48 | 0.18 s / 0.20 s | $0.015 | GPU pod (H100) |
| 31 | system-onedetailsBase model: Qwen3-8Bsource | 53.5 | 47.4 | 63 / 57 | 50 / 40 | 48 / 46 | 0.23 s / 0.36 s | $0.068 | GPU pod (RTX5090) |
| 32 | system-one-openAPIdetailsBase model: Gemma 4 E2Bsource | 44.4 | 38.8 | 60 / 52 | 15 / 15 | 58 / 50 | 1.2 s / 1.3 s | $0.011 | author demo endpoint |
| 33 | Autoloops – Gemma 4 31B ITAPIdetailsBase model: undisclosed | 76.9 | 76.5 | 86 / 88 | 69 / 67 | 76 / 75 | 0.61 s / 0.67 s | tariff$0.103 | hosted API |
| 34 | GPT-6 Luna (low reasoning effort)APIdetailsBase model: undisclosed | 94.7 | 95.9 | 99 / 98 | 85 / 90 | 100 / 100 | 1.6 s / 3.0 s | $0.108 | hosted API |
| 35 | GPT-6 Luna (default medium reasoning effort)APIdetailsBase model: undisclosed | 96.3 | 96.2 | 99 / 100 | 91 / 90 | 99 / 99 | 1.6 s / 3.1 s | $0.114 | hosted API |
| 36 | JevOnedetailsBase model: Qwen/Qwen3.6-35B-A3Bsource | 53.9 | 54.4 | 82 / 77 | 9 / 17 | 70 / 69 | 0.27 s / 0.34 s | $0.101 | GPU pod (RTXPRO6000) |
| 37 | kev 4BdetailsBase model: Qwen/Qwen3-4B-Basesource · Evaluated Qwen3 variant; later releases use a different base. | 47.1 | 33.2 | 54 / 47 | 29 / 4 | 58 / 49 | 0.49 s / 0.57 s | $0.014 | GPU pod (A6000) |
| 38 | kev 8BdetailsBase model: Qwen/Qwen3-8B-Basesource | 54.3 | 42.3 | 61 / 59 | 38 / 14 | 64 / 54 | 0.51 s / 0.75 s | $0.097 | GPU pod (A6000) |
| 39 | open-alternative-jevdetailsBase model: Qwen3.5-4Bsource | 42.6 | 32.1 | 63 / 65 | 5 / -17 | 59 / 48 | 0.22 s / 0.33 s | $0.017 | GPU pod (RTX5090) |
| 40 | Bespoke Nimble 9BdetailsBase model: Qwen3.5-9Bsource | 65.1 | 62.3 | 72 / 73 | 53 / 45 | 71 / 70 | 0.55 s / 0.93 s | $0.128 | GPU pod (A6000) |
| 41 | Malkuth-2BdetailsBase model: empero-ai/Qwen3.8-2B-Distillsource | 44.2 | 28.6 | 55 / 48 | 26 / -6 | 52 / 43 | 0.23 s / 0.29 s | $0.014 | GPU pod (RTX5090) |
| 42 | openjev-sglangAPIdetailsBase model: Qwen/Qwen3.6-35B-A3Bsource | 61.4 | 55.9 | 81 / 75 | 36 / 27 | 67 / 66 | 1.2 s / 1.3 s | $0.140 | author demo endpoint |
| 43 | decider-35b-a3bdetailsBase model: Qwen/Qwen3.5-35B-A3B-Basesource | 64.1 | 56.8 | 74 / 76 | 51 / 31 | 67 / 63 | 0.24 s / 0.33 s | $0.154 | GPU pod (H100) |
| 44 | local-jev Qwen3.5-4BdetailsBase model: Qwen/Qwen3.5-4Bsource | 36.7 | 30.8 | 74 / 69 | -23 / -38 | 60 / 62 | 0.43 s / 0.99 s | $0.023 | GPU pod (A6000) |
| 45 | Nemotron Diffusion 8Bv1.5 roster addendum A3detailsBase model: nvidia/Nemotron-Labs-Diffusion-8Bsource | 39.5 | 28.1 | 57 / 59 | 11 / -17 | 51 / 43 | 0.19 s / 0.23 s | $0.030 | GPU pod (H100) |
| 46 | Open-Jev 9BdetailsBase model: Qwen/Qwen3.5-9Bsource | 61.8 | 65.9 | 66 / 71 | 49 / 59 | 70 / 67 | 1.3 s / 3.4 s | $0.170 | GPU pod (H100) |
| 47 | Decision 2BdetailsBase model: openbmb/MiniCPM5-2Bsource | 34.6 | 28.0 | 58 / 68 | -16 / -35 | 62 / 51 | 0.30 s / 0.31 s | $0.013 | GPU pod (RTX6000) |
| 48 | GPT-5.6 LunaAPIdetailsBase model: undisclosed | 93.2 | 95.5 | 96 / 94 | 87 / 94 | 96 / 99 | 1.3 s / 3.0 s | tariff$0.205 | hosted API |
| 49 | typecastlmdetailsBase model: Qwen3.5-4Bsource | 23.7 | 38.8 | 66 / 72 | -52 / -7 | 57 / 52 | 0.23 s / 0.28 s | $0.016 | GPU pod (RTX5090) |
| 50 | JEV Qwen3.5-9B Base NVFP4detailsBase model: ig1/Qwen3.5-9B-NVFP4source | 32.9 | 31.6 | 71 / 60 | -35 / -22 | 62 / 57 | 0.19 s / 0.23 s | $0.056 | GPU pod (RTX5090) |
| 51 | Gemini 3.1 Flash-LiteAPIdetailsBase model: undisclosed | 76.6 | 78.6 | 84 / 84 | 74 / 75 | 72 / 77 | 0.86 s / 1.1 s | tariff$0.219 | hosted API |
| 52 | AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1Base model: Qwen/Qwen3.8-27Bsource | 71.9 | 73.6 | 85 / 82 | 50 / 62 | 80 / 77 | 0.35 s / 0.49 s | $0.226 | GPU pod (H100) |
| 53 | AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2Base model: Qwen/Qwen3.8-27Bsource | 72.0 | 73.5 | 85 / 82 | 51 / 62 | 80 / 77 | 0.33 s / 0.60 s | $0.226 | GPU pod (RTX PRO 6000) |
| 54 | NInfer Qwen3.8-27B NVFP4detailsBase model: Qwen3.8-27Bsource | 67.0 | 64.0 | 81 / 80 | 48 / 45 | 72 / 67 | 0.25 s / 0.42 s | $0.231 | GPU pod (RTX5090) |
| 55 | Eikos-27Bv1.5 roster addendum A1Base model: Qwen/Qwen3.8-27Bsource | 74.1 | 76.0 | 88 / 88 | 56 / 64 | 78 / 76 | 0.35 s / 0.49 s | $0.238 | GPU pod (H100) |
| 56 | NInfer Qwen3.8-27B NVFP4 (T=1.5)detailsBase model: Qwen3.8-27Bsource | 62.4 | 59.7 | 81 / 80 | 37 / 36 | 70 / 63 | 0.25 s / 0.42 s | $0.231 | GPU pod (RTX5090) |
| 57 | InstinctAPIdetailsBase model: Qwen3.8-27Bsource · Operator-reported; weights not publicly verifiable. | 61.9 | 63.6 | 84 / 80 | 29 / 40 | 73 / 72 | 0.52 s / 1.3 s | $0.230 | author demo endpoint |
| 58 | OpenJev (thinking, BF16)detailsBase model: undisclosedsource | 82.5 | 85.9 | 82 / 79 | 81 / 89 | 85 / 91 | 1.6 s / 2.6 s | $0.241 | GPU pod (H100) |
| 59 | djev (thinking)detailsBase model: google/diffusiongemma-26B-A4B-itsource | 76.6 | 78.1 | 65 / 48 | 75 / 96 | 90 / 90 | 1.5 s / 4.0 s | $0.249 | GPU pod (H100) |
| 60 | LitJevdetailsBase model: Qwen/Qwen3.8-27Bsource | 58.1 | 58.6 | 80 / 79 | 23 / 34 | 70 / 63 | 3.1 s / 4.9 s | $0.244 | GPU pod (H100) |
| 61 | Bev / Bonsai 27Bv1.5 roster addendum A4detailsBase model: Qwen/Qwen3.8-27Bsource | 48.7 | 57.5 | 74 / 76 | 4 / 29 | 68 / 67 | 2.2 s / 2.4 s | $0.247 | GPU pod (H100) |
| 62 | Raw Phi-4 mini direct logitsdetailsBase model: undisclosedsource | 38.3 | 20.0 | 54 / 57 | 6 / -44 | 55 / 47 | 0.28 s / 0.43 s | $0.036 | GPU pod (A6000) |
| 63 | OpenSourceJevdetailsBase model: Qwen3.5-4Bsource | 27.9 | 23.7 | 64 / 51 | -36 / -37 | 55 / 57 | 1.4 s / 4.0 s | $0.011 | GPU pod (A6000) |
| 64 | reflex-27bdetailsBase model: Qwen3.8-27Bsource | 63.1 | 62.5 | 85 / 80 | 31 / 36 | 74 / 71 | 2.7 s / 4.4 s | $0.297 | GPU pod (H100) |
| 65 | Open-Jev 2BdetailsBase model: Qwen/Qwen3.5-2Bsource | 38.8 | 28.3 | 52 / 45 | 17 / -8 | 48 / 48 | 1.0 s / 2.6 s | $0.170 | GPU pod (H100) |
| 66 | GLiNER2 largedetailsBase model: microsoft/deberta-v3-largesource | 26.0 | 19.0 | 27 / 28 | 16 / -1 | 35 / 29 | 1.9 s / 17.7 s | $0.0056 | CPU container |
| 67 | Qwen3-Reranker-4BdetailsBase model: Qwen/Qwen3-4B-Basesource | 30.3 | 13.4 | 57 / 46 | -10 / -46 | 44 / 40 | 0.51 s / 2.1 s | $0.052 | GPU pod (A6000) |
| 68 | DeepSeek V4.1 FlashAPIdetailsBase model: undisclosed | 91.8 | 95.5 | 96 / 98 | 86 / 97 | 93 / 92 | 1.8 s / 6.5 s | $0.498 | hosted API |
| 69 | SimpleJevdetailsBase model: Qwen/Qwen3.5-0.8Bsource | 13.3 | 20.2 | 28 / 28 | -13 / 4 | 25 / 28 | 7.4 s / 16.8 s | $0.0098 | CPU container |
| 70 | SimpleJev Qwen3.8-27BAPIdetailsBase model: Qwen/Qwen3.8-27Bsource | 71.9 | 73.6 | 85 / 81 | 51 / 61 | 79 / 79 | 1.7 s / 1.9 s | $0.687 | author demo endpoint |
| 71 | decision-machine-1APIdetailsBase model: undisclosed | 27.5 | 5.1 | 53 / 45 | -18 / -68 | 47 / 38 | 0.18 s / 0.29 s | $0.029 | hosted API |
| 72 | GLiNER2.5 multidetailsBase model: mDeBERTa-v3-basesource | 15.8 | 12.2 | 19 / 17 | 0 / -3 | 28 / 22 | 1.4 s / 16.1 s | $0.0028 | CPU container |
| 73 | Bosun v3.1 0.6Bv1.5 roster addendum A4detailsBase model: Qwen/Qwen3-0.6Bsource | 23.2 | 5.3 | 33 / 37 | -4 / -58 | 40 / 37 | 4.0 s / 16.5 s | $0.0056 | CPU (shared host) |
| 74 | GLiNER2detailsBase model: DeBERTa-v3-basesource | 17.2 | 9.7 | 25 / 21 | 2 / -10 | 25 / 18 | 1.00 s / 9.3 s | $0.0028 | CPU container |
| 75 | Deem 0.8B v1v1.5 roster addendum A4detailsBase model: Qwen/Qwen3.5-0.8Bsource | 24.7 | 3.6 | 28 / 20 | 8 / -29 | 38 / 20 | 0.50 s / 0.74 s | $0.0045 | GPU pod (H100) |
| 76 | JevActAPIdetailsBase model: undisclosedsource | 19.7 | 3.5 | 31 / 35 | -11 / -55 | 39 / 30 | 0.80 s / 2.8 s | $0.011 | author demo endpoint |
| 77 | CLM-8BdetailsBase model: Qwen/Qwen3-8Bsource | 17.1 | 5.8 | 19 / 8 | 7 / -13 | 25 / 23 | 0.18 s / 0.26 s | $0.045 | GPU pod (RTXPRO6000) |
| 78 | kev 0.6BdetailsBase model: Qwen/Qwen3-0.6B-Basesource | 25.5 | -1.5 | 33 / 21 | 2 / -58 | 41 / 32 | 0.42 s / 0.45 s | $0.0046 | GPU pod (A6000) |
| 79 | Raw Qwen3 0.6B direct logitsdetailsBase model: Qwen/Qwen3-0.6B-Basesource | 10.0 | 11.1 | 17 / 24 | 13 / -4 | -1 / 14 | 0.28 s / 0.33 s | $0.0056 | GPU pod (A6000) |
| 80 | GLiNER2.5 smalldetailsBase model: microsoft/deberta-v3-xsmallsource | 17.2 | 1.6 | 16 / 12 | 4 / -34 | 31 / 27 | 0.47 s / 3.9 s | $0.0028 | CPU container |
| 81 | Raw Qwen3 1.7B direct logitsdetailsBase model: Qwen/Qwen3-1.7B-Basesource | 12.3 | 7.1 | 41 / 23 | 12 / -3 | -16 / 1 | 0.28 s / 0.34 s | $0.011 | GPU pod (A6000) |
| 82 | MirrordetailsBase model: undisclosed | 6.7 | 4.4 | 2 / -9 | -1 / -5 | 19 / 27 | 4.6 s / 9.4 s | $0.0023 | CPU container |
| 83 | ZeroEntropy zerank-2detailsBase model: Qwen/Qwen3-4Bsource | 10.0 | -0.4 | 57 / 47 | -68 / -86 | 41 / 37 | 0.47 s / 2.0 s | $0.052 | GPU pod (A6000) |
| 84 | jeffdetailsBase model: undisclosedsource | 12.7 | -3.6 | 33 / 22 | -28 / -61 | 33 / 28 | 7.1 s / 37.2 s | $0.0043 | CPU container |
| 85 | smalljev semantic-v9detailsBase model: openbmb/MiniCPM5-2B-Basesource | 10.9 | -3.5 | 31 / 18 | -40 / -64 | 42 / 35 | 0.45 s / 0.50 s | $0.020 | GPU pod (A6000) |
| 86 | Laya multilingualv1.5 roster addendum A4detailsBase model: mmBERT-basesource | 2.8 | 1.9 | 20 / 9 | -36 / -31 | 24 / 28 | 1.0 s / 4.1 s | $0.0039 | CPU (shared host) |
| 87 | OpenDecisiondetailsBase model: MoritzLaurer/ModernBERT-large-zeroshot-v2.0source | 9.5 | -5.3 | 34 / 20 | -47 / -71 | 41 / 35 | 0.30 s / 0.74 s | $0.0050 | GPU pod (H100) |
| 88 | BAAI bge-reranker-v2-m3detailsBase model: BAAI/bge-m3source | -21.1 | -22.8 | 4 / 4 | -100 / -100 | 32 / 27 | 0.21 s / 0.42 s | $0.023 | GPU pod (A6000) |
| 89 | Certo v1detailsBase model: ModernBERT-largesource | -24.1 | -24.9 | -2 / -2 | -100 / -100 | 29 / 27 | 0.26 s / 0.27 s | $0.0013 | GPU pod (A6000) |
| 90 | Decision FastdetailsBase model: Qwen/Qwen3-0.6B-Basesource | 7.3 | -11.7 | 38 / 18 | -59 / -87 | 43 / 33 | 0.27 s / 0.27 s | $0.0046 | GPU pod (RTX6000) |
| 91 | Alibaba GTE Reranker ModernBERT-basedetailsBase model: answerdotai/ModernBERT-basesource | -18.6 | -23.2 | 13 / 2 | -100 / -100 | 31 / 28 | 0.21 s / 0.34 s | $0.011 | GPU pod (A6000) |
| 92 | kev 0.5BdetailsBase model: Qwen/Qwen2.5-0.5Bsource | 3.4 | -22.3 | 25 / 11 | -48 / -95 | 33 / 17 | 0.36 s / 0.40 s | $0.0046 | GPU pod (A6000) |
| 93 | LayadetailsBase model: ModernBERT-largesource | -0.5 | -13.2 | 37 / 13 | -76 / -82 | 37 / 29 | 1.5 s / 2.7 s | $0.0032 | CPU container |
| 94 | lev-350mdetailsBase model: LiquidAI/LFM2.5-350Msource | 5.5 | -18.0 | 33 / 15 | -52 / -100 | 36 / 31 | 0.19 s / 0.22 s | $0.0046 | GPU pod (RTX6000) |
| 95 | Qwen3.5-0.8B Decision ModeldetailsBase model: Qwen/Qwen3.5-0.8B-Basesource | 3.4 | -3.7 | 35 / 40 | -70 / -86 | 45 / 34 | 1.3 s / 5.0 s | $0.0048 | CPU container |
| 96 | Mixedbread mxbai-rerank-base-v2detailsBase model: undisclosedsource | -21.0 | -24.0 | 6 / 1 | -100 / -100 | 31 / 27 | 0.23 s / 0.49 s | $0.021 | GPU pod (A6000) |
| 97 | Needle 3detailsBase model: undisclosedsource | -11.1 | -21.1 | -4 / -8 | -15 / -24 | -15 / -32 | 135.5 s / 285.4 s | $0.019 | CPU container |
| 98 | Needle 3, options as toolsdetailsBase model: undisclosedsource | -5.8 | -19.9 | 11 / 2 | -12 / -30 | -16 / -32 | 58.2 s / 136.7 s | $0.019 | CPU container |
| 99 | open-jev-deberta-v3-largedetailsBase model: microsoft/deberta-v3-largesource | -9.3 | -14.6 | 21 / 17 | -79 / -87 | 30 / 27 | 2.9 s / 5.1 s | $0.0056 | CPU container |
| 100 | Open Jev JSON CanvasdetailsBase model: google/diffusiongemma-26B-A4B-itsource | 79.2 | 75.1 | 81 / 76 | 75 / 74 | 81 / 75 | 0.44 s / 0.63 s | $0.049 | GPU pod (H100) |
| 101 | openJev VerdictdetailsBase model: undisclosedsource | 12.6 | -14.1 | 27 / 17 | -15 / -79 | 26 / 20 | 0.40 s / 1.0 s | $0.0028 | CPU container |
| 102 | openJev Verdict 1.4detailsBase model: knowledgator/gliclass-modern-base-v2.0source | -14.7 | -18.3 | 24 / 18 | -100 / -100 | 31 / 27 | 0.77 s / 1.1 s | $0.0028 | CPU container |
| 103 | Qwen3.8 27BAPIdetailsBase model: undisclosed | 92.9 | 98.2 | 97 / 98 | 87 / 99 | 95 / 98 | 6.5 s / 31.9 s | $2.18 | hosted API |
| 104 | verdict-smalldetailsBase model: intfloat/multilingual-e5-smallsource | -0.9 | -2.9 | 26 / 13 | -55 / -44 | 26 / 22 | 0.22 s / 2.9 s | $0.0009 | CPU container |
| 105 | VondetailsBase model: answerdotai/ModernBERT-largesource | -1.9 | -16.6 | 34 / 18 | -79 / -98 | 38 / 30 | 0.92 s / 2.9 s | $0.0038 | CPU container |
| 106 | Laya typed-decisionsv1.5 roster addendum A4detailsBase model: ModernBERT-largesource | -8.5 | -16.9 | 31 / 18 | -92 / -99 | 35 / 31 | 3.3 s / 13.8 s | $0.0038 | CPU (shared host) |
| – | classifier.devAPIhonorable mentiondetailsBase model: undisclosed | 74.5 | 77.1 | 84 / 85 | 57 / 64 | 82 / 82 | 0.52 s / 1.2 s | $0.023 | hosted API |
| – | SimpleJev Qwen3.6-35B-A3BAPIpartial rundetailsBase model: undisclosedsource | -17.4 | -23.0 | 16 / 9 | -36 / -43 | -32 / -35 | 1.7 s / 1.8 s | $0.145 | author demo endpoint |
| – | Decision-4Bv1.5 roster addendum A4not rankeddetailsBase model: Qwen/Qwen3.5-4Bsource | — | — | – / – | – / – | – / – | 0.23 s / 0.38 s | $0.014 | GPU pod (H100) |
Costs are estimates (est.) unless marked tariff.
API = the operator's endpoint received sealed item text during evaluation, without answers. Sealed item text, answers and item-level results stay private; only system-level aggregates appear here. Hover a cost for its price basis and a latency for its raw values and adjustment.
JevBench v1.5.4 · headline option A
Official (A)weighted harmonic mean of four 0–100 axes, Intelligence · Calibration · Speed · Cost = 25 · 25 · 25 · 25, with the low-axis gates · Method ↓
Cygnet and Winnow-12B Q8 are joint leaders (statistical tie).
Whiskers are 95% bootstrap intervals. 69 of 77 adjacent pairs with published paired-bootstrap comparisons are statistical ties — read the order as a ranking, not the gaps as significant. No paired comparison is published for the other 28 adjacent pairs, so no tie classification is inferred.
Costs are estimates (est.) unless marked tariff.
system-one-openJev rebuildJev (TypeSafe, closed)UnclassifiedRaw-logit control (base model)Native-logit decision engineClosed decision APIInstruction model, JSON schemaZero-shot classifierReranker (neutral adapter)Small tool-calling model
A is the official headline: equal 25/25/25/25 axis weights and an Intelligence floor of 50. B remains the secondary 40/20/20/20 axis-weight view; C retains equal axes with an Intelligence floor of 60. All three use equal Choice/Noul/Score weights. The CI column is the paired-bootstrap 95% interval of the A score.
| #A | System | A · equal (headline) | B · 40/20/20/20 | C · equal, I floor 60 | #B | #C | A 95% CI |
|---|---|---|---|---|---|---|---|
| 1 | CygnetdetailsBase model: google/gemma-4-12B-itsource | 73.7 | 73.2 | 73.7 | 2 | 1 | 72.4–74.5 |
| 2 | Winnow-12B Q8detailsBase model: google/gemma-4-12B-itsource | 73.2 | 73.5 | 73.2 | 1 | 2 | 72.0–74.0 |
| 3 | Jev 1.13.0APIdetailsBase model: undisclosed | 72.1 | 72.1 | 72.1 | 3 | 3 | 71.0–72.6 |
| 4 | JevK5 v0.3v1.5 roster addendum A1Base model: Qwen/Qwen3.5-4Bsource | 71.9 | 68.1 | 63.2 | 5 | 8 | 69.4–72.9 |
| 5 | Plumb-4Bv1.5 roster addendum A1Base model: alibiserikbay/JevK5source | 71.6 | 67.7 | 62.0 | 6 | 9 | 69.2–72.7 |
| 6 | Jev-OmnidetailsBase model: google/gemma-4-12B-itsource | 71.5 | 71.3 | 71.5 | 4 | 4 | 70.2–72.4 |
| 7 | decider-4b v2detailsBase model: Qwen/Qwen3.5-4B-Basesource | 71.3 | 67.5 | 61.6 | 7 | 10 | 69.1–72.3 |
| 8 | Decision 4B v1.2v1.5 roster addendum A1Base model: undisclosed | 70.8 | 66.6 | 56.7 | 9 | 12 | 68.5–72.0 |
| 9 | Imajev-4Bv1.5 roster addendum A2Base model: Qwen/Qwen3.5-4Bsource | 70.4 | 66.2 | 55.9 | 11 | 13 | 67.8–71.6 |
| 10 | Decision 4B v1.1v1.5 roster addendum A1Base model: undisclosed | 70.4 | 66.1 | 55.1 | 12 | 14 | 66.9–71.6 |
| 11 | Manchego v2.1v1.5 roster addendum A5detailsBase model: Qwen/Qwen3.5-4Bsource | 68.8 | 64.4 | 50.1 | 14 | 20 | 59.6–70.3 |
| 12 | SemIfdetailsBase model: Qwen/Qwen3.5-4Bsource | 68.7 | 64.3 | 50.2 | 15 | 19 | 60.1–70.2 |
| 13 | spark-s1-4b-v6detailsBase model: Qwen3.5-4Bsource | 68.2 | 66.9 | 68.2 | 8 | 5 | 66.3–69.7 |
| 14 | metask-jev-4bdetailsBase model: Qwen3.5-4Bsource | 67.5 | 64.1 | 53.6 | 16 | 16 | 65.4–68.7 |
| 15 | HopperdetailsBase model: Qwen/Qwen3.5-4Bsource | 67.5 | 62.9 | 46.9 | 19 | 24 | 56.3–69.2 |
| 16 | Malkuth-4BdetailsBase model: Qwen/Qwen3.5-4B-Basesource | 66.8 | 63.9 | 55.0 | 17 | 15 | 64.9–67.9 |
| 17 | Surogate Rune 26B-A4B v3v1.5 roster addendum A2Base model: google/gemma-4-26B-A4B-itsource | 66.5 | 66.6 | 66.5 | 10 | 6 | 65.4–67.0 |
| 18 | reflex 4BdetailsBase model: Qwen/Qwen3.5-4Bsource | 65.2 | 61.9 | 48.0 | 20 | 21 | 58.6–66.3 |
| 19 | jev-localdetailsBase model: Qwen/Qwen3.5-9Bsource | 65.2 | 63.2 | 57.4 | 18 | 11 | 63.5–66.5 |
| 20 | djev (Maisa, diffusion-gemma)detailsBase model: google/diffusiongemma-26B-A4B-itsource | 64.2 | 64.8 | 64.2 | 13 | 7 | 63.1–65.0 |
| 21 | Raw Qwen3 4B Instruct 2507 direct logitsdetailsBase model: undisclosedsource | 62.1 | 60.3 | 50.4 | 21 | 18 | 59.4–64.2 |
| 22 | jqvdetailsBase model: Qwen3-32Bsource | 60.8 | 57.5 | 42.2 | 22 | 26 | 51.3–64.3 |
| 23 | JevK5 v0.2.0detailsBase model: Qwen3.5-4Bsource | 58.1 | 53.6 | 40.4 | 23 | 29 | 47.6–68.2 |
| 24 | Qwen3.5-9B Jev-like data-mix v2detailsBase model: Qwen/Qwen3.5-9Bsource | 53.0 | 52.5 | 53.0 | 24 | 17 | 51.9–53.9 |
| 25 | Standard One 8BdetailsBase model: mistralai/Ministral-3-8B-Instruct-2512-BF16source | 47.8 | 47.1 | 47.2 | 27 | 23 | 46.7–48.5 |
| 26 | NInfer Qwen3.8-Flash-Next mixeddetailsBase model: Qwen3.8-Flash-Nextsource | 47.5 | 47.7 | 47.5 | 25 | 22 | 46.7–47.9 |
| 27 | Instinct Dual 4Bv1.5 roster addendum A4APIdetailsBase model: Qwen3.5-4Bsource · Operator-reported; weights not publicly verifiable. | 47.0 | 43.0 | 32.6 | 31 | 32 | 37.9–56.8 |
| 28 | swanOnedetailsBase model: Mia-AiLab/Qwen3.8-Flash-Next-NVFP4source | 46.6 | 47.3 | 46.6 | 26 | 25 | 45.8–46.9 |
| 29 | Raw Qwen3 8B direct logitsdetailsBase model: Qwen/Qwen3-8B-Basesource | 45.2 | 44.6 | 32.9 | 28 | 31 | 36.9–46.7 |
| 30 | decider-2bdetailsBase model: Qwen/Qwen3.5-2B-Basesource | 45.1 | 41.1 | 31.3 | 34 | 34 | 33.8–54.3 |
| 31 | system-onedetailsBase model: Qwen3-8Bsource | 44.1 | 43.4 | 31.2 | 29 | 35 | 36.8–45.7 |
| 32 | system-one-openAPIdetailsBase model: Gemma 4 E2Bsource | 42.4 | 38.8 | 29.5 | 35 | 37 | 33.8–51.9 |
| 33 | Autoloops – Gemma 4 31B ITAPIdetailsBase model: undisclosed | 40.5 | 41.8 | 40.5 | 32 | 27 | 40.1–40.8 |
| 34 | GPT-6 Luna (low reasoning effort)APIdetailsBase model: undisclosed | 40.5 | 43.1 | 40.5 | 30 | 28 | 40.3–40.7 |
| 35 | GPT-6 Luna (default medium reasoning effort)APIdetailsBase model: undisclosed | 38.8 | 41.4 | 38.8 | 33 | 30 | 38.6–38.9 |
| 36 | JevOnedetailsBase model: Qwen/Qwen3.6-35B-A3Bsource | 38.2 | 37.4 | 31.1 | 36 | 36 | 37.4–38.8 |
| 37 | kev 4BdetailsBase model: Qwen/Qwen3-4B-Basesource · Evaluated Qwen3 variant; later releases use a different base. | 38.1 | 34.6 | 26.4 | 37 | 40 | 27.5–46.8 |
| 38 | kev 8BdetailsBase model: Qwen/Qwen3-8B-Basesource | 34.2 | 33.1 | 23.7 | 38 | 42 | 26.8–37.5 |
| 39 | open-alternative-jevdetailsBase model: Qwen3.5-4Bsource | 33.6 | 29.9 | 23.3 | 40 | 43 | 25.8–41.7 |
| 40 | Bespoke Nimble 9BdetailsBase model: Qwen3.5-9Bsource | 31.8 | 32.3 | 31.8 | 39 | 33 | 31.2–32.3 |
| 41 | Malkuth-2BdetailsBase model: empero-ai/Qwen3.8-2B-Distillsource | 29.9 | 26.4 | 20.8 | 43 | 45 | 21.0–39.0 |
| 42 | openjev-sglangAPIdetailsBase model: Qwen/Qwen3.6-35B-A3Bsource | 29.0 | 29.2 | 27.7 | 41 | 38 | 28.4–29.4 |
| 43 | decider-35b-a3bdetailsBase model: Qwen/Qwen3.5-35B-A3B-Basesource | 27.5 | 27.7 | 27.5 | 42 | 39 | 26.9–27.9 |
| 44 | local-jev Qwen3.5-4BdetailsBase model: Qwen/Qwen3.5-4Bsource | 25.8 | 22.7 | 17.9 | 46 | 54 | 19.9–33.0 |
| 45 | Nemotron Diffusion 8Bv1.5 roster addendum A3detailsBase model: nvidia/Nemotron-Labs-Diffusion-8Bsource | 25.7 | 22.7 | 17.8 | 47 | 55 | 18.6–32.8 |
| 46 | Open-Jev 9BdetailsBase model: Qwen/Qwen3.5-9Bsource | 24.4 | 25.0 | 24.4 | 44 | 41 | 23.9–24.6 |
| 47 | Decision 2BdetailsBase model: openbmb/MiniCPM5-2Bsource | 22.5 | 19.3 | 15.6 | 53 | 57 | 16.7–28.8 |
| 48 | GPT-5.6 LunaAPIdetailsBase model: undisclosed | 22.4 | 24.2 | 22.4 | 45 | 44 | 22.3–22.5 |
| 49 | typecastlmdetailsBase model: Qwen3.5-4Bsource | 21.8 | 18.9 | 15.2 | 57 | 59 | 16.2–28.6 |
| 50 | JEV Qwen3.5-9B Base NVFP4detailsBase model: ig1/Qwen3.5-9B-NVFP4source | 20.1 | 17.8 | 13.9 | 59 | 60 | 15.2–25.5 |
| 51 | Gemini 3.1 Flash-LiteAPIdetailsBase model: undisclosed | 19.6 | 20.8 | 19.6 | 48 | 46 | 19.3–19.8 |
| 52 | AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1Base model: Qwen/Qwen3.8-27Bsource | 19.5 | 20.4 | 19.5 | 49 | 47 | 19.3–19.6 |
| 53 | AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2Base model: Qwen/Qwen3.8-27Bsource | 19.5 | 20.4 | 19.5 | 50 | 48 | 19.3–19.6 |
| 54 | NInfer Qwen3.8-27B NVFP4detailsBase model: Qwen3.8-27Bsource | 18.7 | 19.3 | 18.7 | 52 | 49 | 18.5–18.9 |
| 55 | Eikos-27Bv1.5 roster addendum A1Base model: Qwen/Qwen3.8-27Bsource | 18.5 | 19.5 | 18.5 | 51 | 50 | 18.3–18.6 |
| 56 | NInfer Qwen3.8-27B NVFP4 (T=1.5)detailsBase model: Qwen3.8-27Bsource | 18.5 | 18.9 | 18.5 | 55 | 51 | 18.2–18.6 |
| 57 | InstinctAPIdetailsBase model: Qwen3.8-27Bsource · Operator-reported; weights not publicly verifiable. | 18.3 | 18.9 | 18.3 | 56 | 52 | 18.0–18.5 |
| 58 | OpenJev (thinking, BF16)detailsBase model: undisclosedsource | 17.9 | 19.3 | 17.9 | 54 | 53 | 17.8–18.1 |
| 59 | djev (thinking)detailsBase model: google/diffusiongemma-26B-A4B-itsource | 17.4 | 18.5 | 17.4 | 58 | 56 | 17.3–17.5 |
| 60 | LitJevdetailsBase model: Qwen/Qwen3.8-27Bsource | 16.3 | 16.7 | 15.4 | 60 | 58 | 16.0–16.5 |
| 61 | Bev / Bonsai 27Bv1.5 roster addendum A4detailsBase model: Qwen/Qwen3.8-27Bsource | 15.8 | 16.0 | 12.3 | 61 | 62 | 15.1–16.0 |
| 62 | Raw Phi-4 mini direct logitsdetailsBase model: undisclosedsource | 15.2 | 13.1 | 10.6 | 63 | 63 | 10.3–20.7 |
| 63 | OpenSourceJevdetailsBase model: Qwen3.5-4Bsource | 13.2 | 11.2 | 9.2 | 64 | 64 | 9.0–18.3 |
| 64 | reflex-27bdetailsBase model: Qwen3.8-27Bsource | 13.2 | 13.8 | 13.2 | 62 | 61 | 13.0–13.3 |
| 65 | Open-Jev 2BdetailsBase model: Qwen/Qwen3.5-2Bsource | 9.1 | 8.4 | 6.3 | 65 | 66 | 6.6–11.5 |
| 66 | GLiNER2 largedetailsBase model: microsoft/deberta-v3-largesource | 8.4 | 7.2 | 5.8 | 67 | 67 | 5.2–12.3 |
| 67 | Qwen3-Reranker-4BdetailsBase model: Qwen/Qwen3-4B-Basesource | 7.1 | 5.9 | 4.9 | 68 | 68 | 4.3–10.7 |
| 68 | DeepSeek V4.1 FlashAPIdetailsBase model: undisclosed | 6.6 | 7.4 | 6.6 | 66 | 65 | 6.6–6.7 |
| 69 | SimpleJevdetailsBase model: Qwen/Qwen3.5-0.8Bsource | 4.0 | 3.3 | 2.8 | 70 | 70 | 2.1–6.7 |
| 70 | SimpleJev Qwen3.8-27BAPIdetailsBase model: Qwen/Qwen3.8-27Bsource | 3.4 | 3.7 | 3.4 | 69 | 69 | 3.3–3.4 |
| 71 | decision-machine-1APIdetailsBase model: undisclosed | 3.2 | 2.5 | 2.3 | 71 | 71 | 1.7–5.4 |
| 72 | GLiNER2.5 multidetailsBase model: mDeBERTa-v3-basesource | 2.7 | 2.1 | 1.9 | 72 | 72 | 1.2–5.1 |
| 73 | Bosun v3.1 0.6Bv1.5 roster addendum A4detailsBase model: Qwen/Qwen3-0.6Bsource | 2.5 | 1.9 | 1.7 | 73 | 73 | 1.2–4.6 |
| 74 | GLiNER2detailsBase model: DeBERTa-v3-basesource | 2.3 | 1.8 | 1.6 | 74 | 74 | 0.9–4.5 |
| 75 | Deem 0.8B v1v1.5 roster addendum A4detailsBase model: Qwen/Qwen3.5-0.8Bsource | 2.1 | 1.7 | 1.5 | 75 | 75 | 0.9–4.3 |
| 76 | JevActAPIdetailsBase model: undisclosedsource | 1.5 | 1.1 | 1.1 | 77 | 76 | 0.5–3.2 |
| 77 | CLM-8BdetailsBase model: Qwen/Qwen3-8Bsource | 1.5 | 1.2 | 1.0 | 76 | 77 | 0.5–3.2 |
| 78 | kev 0.6BdetailsBase model: Qwen/Qwen3-0.6B-Basesource | 1.3 | 0.9 | 0.9 | 78 | 78 | 0.5–2.5 |
| 79 | Raw Qwen3 0.6B direct logitsdetailsBase model: Qwen/Qwen3-0.6B-Basesource | 1.1 | 0.9 | 0.7 | 79 | 79 | 0.3–2.6 |
| 80 | GLiNER2.5 smalldetailsBase model: microsoft/deberta-v3-xsmallsource | 0.9 | 0.6 | 0.6 | 81 | 80 | 0.2–2.2 |
| 81 | Raw Qwen3 1.7B direct logitsdetailsBase model: Qwen/Qwen3-1.7B-Basesource | 0.9 | 0.7 | 0.6 | 80 | 81 | 0.1–2.3 |
| 82 | MirrordetailsBase model: undisclosed | 0.2 | 0.2 | 0.2 | 82 | 82 | 0.0–0.9 |
| 83 | ZeroEntropy zerank-2detailsBase model: Qwen/Qwen3-4Bsource | 0.1 | 0.1 | 0.1 | 83 | 83 | 0.0–0.5 |
| 84 | jeffdetailsBase model: undisclosedsource | 0.1 | 0.1 | 0.1 | 84 | 84 | 0.0–0.5 |
| 85 | smalljev semantic-v9detailsBase model: openbmb/MiniCPM5-2B-Basesource | 0.1 | 0.0 | 0.0 | 85 | 85 | 0.0–0.4 |
| 86 | Laya multilingualv1.5 roster addendum A4detailsBase model: mmBERT-basesource | 0.0 | 0.0 | 0.0 | 86 | 86 | 0.0–0.3 |
| 87 | OpenDecisiondetailsBase model: MoritzLaurer/ModernBERT-large-zeroshot-v2.0source | 0.0 | 0.0 | 0.0 | 87 | 87 | 0.0–0.2 |
| 88 | BAAI bge-reranker-v2-m3detailsBase model: BAAI/bge-m3source | 0.0 | 0.0 | 0.0 | 88 | 88 | 0.0–0.0 |
| 89 | Certo v1detailsBase model: ModernBERT-largesource | 0.0 | 0.0 | 0.0 | 89 | 89 | 0.0–0.0 |
| 90 | Decision FastdetailsBase model: Qwen/Qwen3-0.6B-Basesource | 0.0 | 0.0 | 0.0 | 90 | 90 | 0.0–0.0 |
| 91 | Alibaba GTE Reranker ModernBERT-basedetailsBase model: answerdotai/ModernBERT-basesource | 0.0 | 0.0 | 0.0 | 91 | 91 | 0.0–0.0 |
| 92 | kev 0.5BdetailsBase model: Qwen/Qwen2.5-0.5Bsource | 0.0 | 0.0 | 0.0 | 92 | 92 | 0.0–0.0 |
| 93 | LayadetailsBase model: ModernBERT-largesource | 0.0 | 0.0 | 0.0 | 93 | 93 | 0.0–0.0 |
| 94 | lev-350mdetailsBase model: LiquidAI/LFM2.5-350Msource | 0.0 | 0.0 | 0.0 | 94 | 94 | 0.0–0.0 |
| 95 | Qwen3.5-0.8B Decision ModeldetailsBase model: Qwen/Qwen3.5-0.8B-Basesource | 0.0 | 0.0 | 0.0 | 95 | 95 | 0.0–0.0 |
| 96 | Mixedbread mxbai-rerank-base-v2detailsBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 96 | 96 | 0.0–0.0 |
| 97 | Needle 3detailsBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 97 | 97 | 0.0–0.0 |
| 98 | Needle 3, options as toolsdetailsBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 98 | 98 | 0.0–0.0 |
| 99 | open-jev-deberta-v3-largedetailsBase model: microsoft/deberta-v3-largesource | 0.0 | 0.0 | 0.0 | 99 | 99 | 0.0–0.0 |
| 100 | Open Jev JSON CanvasdetailsBase model: google/diffusiongemma-26B-A4B-itsource | 0.0 | 0.0 | 0.0 | 100 | 100 | 0.0–0.0 |
| 101 | openJev VerdictdetailsBase model: undisclosedsource | 0.0 | 0.0 | 0.0 | 101 | 101 | 0.0–0.0 |
| 102 | openJev Verdict 1.4detailsBase model: knowledgator/gliclass-modern-base-v2.0source | 0.0 | 0.0 | 0.0 | 102 | 102 | 0.0–0.0 |
| 103 | Qwen3.8 27BAPIdetailsBase model: undisclosed | 0.0 | 0.0 | 0.0 | 103 | 103 | 0.0–0.0 |
| 104 | verdict-smalldetailsBase model: intfloat/multilingual-e5-smallsource | 0.0 | 0.0 | 0.0 | 104 | 104 | 0.0–0.0 |
| 105 | VondetailsBase model: answerdotai/ModernBERT-largesource | 0.0 | 0.0 | 0.0 | 105 | 105 | 0.0–0.0 |
| 106 | Laya typed-decisionsv1.5 roster addendum A4detailsBase model: ModernBERT-largesource | 0.0 | 0.0 | 0.0 | 106 | 106 | 0.0–0.0 |
The chart and table above compare the tested systems, not marketing claims. These are the practical answers readers most often need before choosing a Jev-class decision model.
The highest-ranked open entrants in this run are Cygnet (#1, 73.7), Winnow-12B Q8 (#2, 73.2), JevK5 v0.3 (#4, 71.9), Plumb-4B (#5, 71.6). “Open” here means the tested row publishes code or weights; check the licence and exact configuration in the board before adopting one.
Open entrants with released code or weights can run on infrastructure you choose, including EU infrastructure. That can support data residency, but neither open source nor an EU server makes a deployment GDPR-compliant by itself. Assess your data, contracts, retention, subprocessors and security for the complete setup. See Benchmark Heaven's broader EU-hosting comparison.
jev-router.com offers self-hosted open decision models. Neutrality disclosure: it is run by the authors of this benchmark; it receives no scoring advantage and is not a ranked entrant.
The official score (option A) is the equal-weight harmonic mean of Intelligence, Calibration, Speed and Cost — 25/25/25/25 — with an Intelligence floor of 50 and low-axis gates on Speed and Cost. Version v1.5.4 measures 904 open and 720 sealed decisions per system; Choice, Noul and Score each carry a third, sealed items contribute 50% of Intelligence, and an open-minus-sealed gap beyond the field median costs points. Method notes · options B and C.
Open an issue in the JevBench repository with a reproducible endpoint or runnable code, the exact model and licence, and whether public JevBench items were used during development. New entrants use the same frozen harness and appear in a new version or a disclosed roster addendum. For private data, see the custom evaluation options.
Every price here is US dollars per 1,000 decisions — not per 1,000 tokens. One decision is a whole typed request — state, rubric and options — not a single token.
Systems with a public tariff (per token or per request) are priced at that tariff times the tokens we measured — 3 rows carry a tariff. Systems without one — open weights, author demos, models we ran ourselves — are priced as if a large inference provider hosted them: the list price of the same weights, or the nearest larger sibling or size class when the exact weights are not listed. We do not use per-minute GPU rental or our own CPU time — providers buy capacity in bulk or own the hardware, and price accordingly. Price × tokens per decision = $ per 1,000 decisions, marked “est.”.
Price rules (v1.5). Only public, bookable list prices that have been in effect for at least 30 days count; a manufacturer's standard, non-promotional launch list price counts from day one, and promotions, subsidies, credits and free tiers never do. The scoring price is never below the market reference price of the system's base model. A system without any eligible price is listed as unpriced — no Cost axis and no score until a price qualifies. A later price change triggers a re-score with a visible note on the row.
API models with a known base model (from 1 Oct 2026). They are ranked at their developer's own stated API price; a striped second bar shows the score and rank they would have at the base-model reference price, the way we price self-served open weights of the same base. First applied on Image JevBench v0.1.5 (Wity-1). The v1.5.4 scores on this page are unchanged; JevBench rows follow this rule from the next version.
All 17 A1/A2/A3/A4/A5 systems below completed all 1,624 decisions and now have official ranks in A, B, and C; the interactive score presets also include them. Their scores, individual 95% intervals, and the frozen v1.5.0 G_med are unchanged. No new paired-bootstrap comparisons were computed for addendum systems. Existing tie markers are retained only for base-system pairs that remain adjacent; no tie or separation is inferred for the other pairs.
| System | Rank (A) | A score · 95% CI | Rank (B) | B score · 95% CI | Rank (C) | C score · 95% CI |
|---|---|---|---|---|---|---|
| JevK5 v0.3v1.5 roster addendum A1 | #4 | 71.9 69.4–72.9 | #5 | 68.1 64.9–69.8 | #8 | 63.2 51.3–72.1 |
| Plumb-4Bv1.5 roster addendum A1 | #5 | 71.6 69.2–72.7 | #6 | 67.7 64.7–69.6 | #9 | 62.0 50.7–70.9 |
| Decision 4B v1.2v1.5 roster addendum A1 | #8 | 70.8 68.5–72.0 | #9 | 66.6 63.7–68.4 | #12 | 56.7 47.6–65.1 |
| Imajev-4Bv1.5 roster addendum A2 | #9 | 70.4 67.8–71.6 | #11 | 66.2 63.1–68.1 | #13 | 55.9 47.1–64.5 |
| Decision 4B v1.1v1.5 roster addendum A1 | #10 | 70.4 66.9–71.6 | #12 | 66.1 62.1–67.9 | #14 | 55.1 46.4–63.5 |
| Manchego v2.1v1.5 roster addendum A5 | #11 | 68.8 59.6–70.3 | #14 | 64.4 55.2–66.5 | #20 | 50.1 41.4–58.1 |
| Surogate Rune 26B-A4B v3v1.5 roster addendum A2 | #17 | 66.5 65.4–67.0 | #10 | 66.6 65.1–67.5 | #6 | 66.5 65.4–67.0 |
| Instinct Dual 4Bv1.5 roster addendum A4API | #27 | 47.0 37.9–56.8 | #31 | 43.0 34.2–52.8 | #32 | 32.6 26.3–39.4 |
| Nemotron Diffusion 8Bv1.5 roster addendum A3 | #45 | 25.7 18.6–32.8 | #47 | 22.7 16.1–29.5 | #55 | 17.8 12.9–22.8 |
| AutoJev-27B (denis-pplx, Qwen3.8-27B)v1.5 roster addendum A1 | #52 | 19.5 19.3–19.6 | #49 | 20.4 20.1–20.6 | #47 | 19.5 19.3–19.6 |
| AutoJev-27B (RTX PRO 6000)v1.5 roster addendum A2 | #53 | 19.5 19.3–19.6 | #50 | 20.4 20.1–20.6 | #48 | 19.5 19.3–19.6 |
| Eikos-27Bv1.5 roster addendum A1 | #55 | 18.5 18.3–18.6 | #51 | 19.5 19.2–19.7 | #50 | 18.5 18.3–18.6 |
| Bev / Bonsai 27Bv1.5 roster addendum A4 | #61 | 15.8 15.1–16.0 | #61 | 16.0 15.2–16.4 | #62 | 12.3 10.5–14.3 |
| Bosun v3.1 0.6Bv1.5 roster addendum A4 | #73 | 2.5 1.2–4.6 | #73 | 1.9 0.9–3.6 | #73 | 1.7 0.8–3.2 |
| Deem 0.8B v1v1.5 roster addendum A4 | #75 | 2.1 0.9–4.3 | #75 | 1.7 0.7–3.5 | #75 | 1.5 0.6–3.0 |
| Laya multilingualv1.5 roster addendum A4 | #86 | 0.0 0.0–0.3 | #86 | 0.0 0.0–0.2 | #86 | 0.0 0.0–0.2 |
| Laya typed-decisionsv1.5 roster addendum A4 | #106 | 0.0 0.0–0.0 | #106 | 0.0 0.0–0.0 | #106 | 0.0 0.0–0.0 |
Ranks follow option scores; score intervals are per system, not pairwise rank comparisons. Every slider preset sorts these same ranked systems using its selected weights.
These systems are part of the 112-system v1.5 roster but have no rank. Their numbers are never shown as zero or free.
Incomplete and unmeasured systems receive no official rank. Existing results from earlier benchmark versions remain on their frozen version pages.
The official score is the weighted harmonic mean of the four axes. Option A gives each axis 25%; option B (40/20/20/20) is a secondary view, while option C keeps equal axes and sets the Intelligence gate at 60. In A and B, Intelligence, Speed and Cost each have a quadratic gate below 50. To limit benchmaxxing on the public items, the open-minus-sealed Intelligence gap may be up to 8 points above the field median (G_med) before a penalty applies. Each further point lowers the multiplier on unpenalized Intelligence by one percentage point.
Frozen method METHOD-v1.5, SHA-256 c25d3d8b8512…; pricing addendum v1.5-M2, SHA-256 2fc44459ef80….
The method owner chose equal axis weights and equal weights for Choice, Noul and Score after reviewing the What-If Lab, preserving continuity with v1.4 and treating the three decision types equally. Disclosed headline amendment: equal-axis, equal-type A, SHA-256 752ddccc4e19…. B remains a secondary view.
Re-evaluation policy. Every release re-evaluates the current top 10 on the composite score. Models ranked #11 and below are re-evaluated on a slower cadence — at least monthly, or with every third scheduled refresh release, whichever comes first — and their score is shown as last measured on its release. A material method change re-evaluates every model. Paid fast-lane runs are evaluated within 48 hours of payment, and new submissions are evaluated in the order received.
base = 0.5 × I_open + 0.5 × I_sealed.excess = gap − G_med, penalty = max(0, 1 − max(0, excess − 8) / 100). G_med for this batch is 5.2 CC points.1b660648bd49….Data file SHA-256 0cf210b76bf85084a5f3fb40fb109e9a2c2f93df42ff6628696377666e89db45 · scorer output SHA-256 452885de2a84cd5b9ed393d541fd8d6c6a540f9d7f762f9d2df9349383f26173 · run kind official.
Harness, public tasks and every scoring rule: github.com/fstandhartinger/jevbench (MIT). Each project links its author's repository or vendor page.
Authors: if we tested the wrong configuration, tell us and we will rerun it. New entrants become a new version or a disclosed roster addendum rather than silently changing this one.
3D view: three.js r128 (MIT).
Loading capability views…
Loading context-length views…
Previous release: JevBench v1.5.3 (frozen results).
Model author? Request a priority evaluation · Submit a model for evaluation →
Open this section to load the earlier public-only board and diagnostics.