Benchmarks
More
OverviewBenchmaxxingCompareChartsCost vs CapabilityProviders per ModelProvider explorerGatewaysEU & SovereignJev-class modelsImage JevBenchAbout

JevBench by Benchmark Heaven · released v1.4.2.2

Jev vs Winnow-12B Q8: published benchmark comparison

A side-by-side view of Jev 1.13.0 and Winnow-12B Q8 from the same hash-checked v1.4.2.2 aggregate. The overall score is a composite; compare the separate measures against your use case.

Four-radar comparison: Jev and Winnow-12B Q8

Four radars compare this fixed pair across the score axes, accuracy per tier, and accuracy by family on the hard tier and sealed set. Further out is better on every spoke.

  • A: Jev 1.13.0 — Jev (TypeSafe, closed) · Score 63.3 (#4)
  • B: Winnow-12B Q8 — Jev rebuild · Score 55.6 (#8)

The four score axes

Radar: the four score axes, two systemsThe four score axes, Jev 1.13.0 vs Winnow-12B Q8. Intelligence: 53.1 vs 48.3; Calibration: 76.3 vs 64.8; Speed: 83.3 vs 82.3; Cost: 52.0 vs 52.9.50100Intelligence53.1 · 48.3Calibration76.3 · 64.8Speed83.3 · 82.3Cost52.0 · 52.9
0–100, the values in the table. A label-only system has no calibration (counted as 0).

Accuracy per tier, incl. sealed

Radar: accuracy per tier, incl. sealed, two systemsAccuracy per tier, incl. sealed, Jev 1.13.0 vs Winnow-12B Q8. Easy: 100% vs 100%; Standard: 99% vs 97%; Judge: 95% vs 91%; Hard: 74% vs 71%; Sealed: 37% vs 33%.50100Easy100% · 100%Standard99% · 97%Judge95% · 91%Hard74% · 71%Sealed37% · 33%
Share correct per tier; Sealed = the 308 private decisions, aggregate only.

Hard tier by family (v1.2 topics)

Radar: hard tier by family (v1.2 topics), two systemsHard tier by family (v1.2 topics), Jev 1.13.0 vs Winnow-12B Q8. Adversarial: 100% vs 92%; Ambiguous: 79% vs 86%; Judge: 79% vs 88%; Long policy: 61% vs 66%; Multi-hop: 86% vs 69%; Probability: 80% vs 55%; Routing: 100% vs 100%; Temporal / numeric: 27% vs 27%; Trade-off: 92% vs 83%; Trap: 100% vs 100%.50100Adversarial100% · 92%Ambiguous79% · 86%Judge79% · 88%Long policy61% · 66%Multi-hop86% · 69%Probability80% · 55%Routing100% · 100%Temporal /numeric27% · 27%Trade-off92% · 83%Trap100% · 100%
Share correct within each family of the 220 v1.2 hard-tier decisions (public and held-out).

Sealed set by family

Radar: sealed set by family, two systemsSealed set by family, Jev 1.13.0 vs Winnow-12B Q8. Ambiguous / abstain: 30% vs 35%; Judge: 34% vs 46%; Long policy: 28% vs 23%; Multi-hop: 45% vs 37%; Paraphrase: 64% vs 64%; Probability: 50% vs 32%; Safety judge: 38% vs 50%; Temporal / numeric: 29% vs 16%; Trade-off: 38% vs 23%; Trap / adversarial: 42% vs 50%.50100Ambiguous /abstain30% · 35%Judge34% · 46%Long policy28% · 23%Multi-hop45% · 37%Paraphrase64% · 64%Probability50% · 32%Safety judge38% · 50%Temporal /numeric29% · 16%Trade-off38% · 23%Trap /adversarial42% · 50%
Share correct within each sealed family — system-level aggregates; the items stay private.

Published values

Jev 1.13.0 (TypeSafe AI) has the higher published JevBench Score. The displayed score uses one decimal place; the comparison above uses the same published release.

All values as a table
MeasureJev 1.13.0 (TypeSafe AI)Winnow-12B Q8
Published rank#4#8
JevBench Score63.355.6
Sealed-set accuracy36.7%33.1%
Intelligence axis53.148.3
Calibration axis76.364.8
Speed axis83.382.3
Cost axis52.052.9
Cost evidencemeasuredestimated
Cost per 1,000 decisions$0.040 per 1,000 decisions$0.037 per 1,000 decisions

Intelligence, Calibration, Speed and Cost — equal-weight harmonic mean, with generalization and Jev-class gates. Cost evidence is labeled per row. Speed is a benchmark axis; deployment latency depends on the endpoint and conditions.

Jev 1.13.0 (TypeSafe AI)

Marked closed in the published row. License note: proprietary API.

Published source

Measured on
production API (api.typesafe.ai)
Speed adjustment
none (production API)
Cost basis
public tariff x measured tokens (https://docs.typesafe.ai/models (output tokens not billed)) [corrected in v1.2.3: the price now averages each of the 314 v1.1 decisions once; see results/v1.2/cost-correction-v1.2.3.json] | public tariff x measured tokens (hard-tier run)

Winnow-12B Q8

Code and weights marked open in the published row. License note: Apache-2.0, including the applicable Gemma 4 base/derivative licence terms.

Published source

Measured on
our GPU (lium.io RTX 4090 24 GB), reached over the internet from Germany; serial, one request at a time
Speed adjustment
x2 + 0.15 s (assumption, not measured)
Cost basis
ESTIMATE: hosted-provider price, OpenRouter google/gemma-3-12b-it hosted reference list price $0.05/M in, $0.0/M out (the nearest publicly hosted 12B Gemma sibling; Winnow reads answer logits in one forward pass and generates no answer tokens) x 393 input and 0 output tokens per decision (input tokens measured (the system's own count))

Compare more JevBench pairs: Jev vs Imajev-4B · Jev vs Plumb-4B · Jev vs decider-4b v2 · Jev vs JevK5 · Jev vs Cygnet · Jev vs Hopper · Jev vs reflex 4B · Jev vs Laya

See Jev alternatives · Choose by use case

Frequently asked questions

What does JevBench show for Jev and Winnow-12B Q8?
In v1.4.2.2, Jev is rank 4 with a JevBench Score of 63.3; Winnow-12B Q8 is rank 8 with a score of 55.6. The table shows their published axes and sealed-set accuracy separately.
Which system has higher sealed-set accuracy?
Jev 1.13.0 (TypeSafe AI) has the higher published sealed-set accuracy (36.7% versus 33.1%). This is separate from the composite JevBench Score.
Can I compare the cost values as actual bills?
Jev 1.13.0 (TypeSafe AI): measured at $0.040 per 1,000 decisions. Winnow-12B Q8: estimated at $0.037 per 1,000 decisions. Estimated and announced bases are not measured charges; inspect the board’s full row disclosure.
Is Winnow-12B Q8 open source, and is Jev?
The published row lists Winnow-12B Q8 with license note “Apache-2.0, including the applicable Gemma 4 base/derivative licence terms”. Jev 1.13.0 is listed as “proprietary API”: TypeSafe AI serves it through its own API and has not published its weights. Check each linked source for the exact terms.