Benchmarks
More
OverviewBenchmaxxingCompareChartsCost vs CapabilityProviders per ModelProvider explorerGatewaysEU & SovereignJev-class modelsImage JevBenchAbout
JevBench by Benchmark Heaven · v1.4.2.2 · individual system

CLM-8B (Contrastive-LM, clm-latest)

system-one-open · by Contrastive-LM (Kwok, Kang, Suresh, Saad-Falcon, Pavone, Ré, Mirhoseini)

JevBench v1.4.2.2 score

8.6

Rank #80 of 91 ranked systems.

54.7 points behind Jev 1.13.0's 63.3.

Where it sits among the 91 ranked systems. The marked tick is Jev 1.13.0 (63.3).

Published axes

intelligence
22.4
calibration
39.8
speed
93.6
cost
78.4

Against Jev 1.13.0

Four radars compare this fixed pair across the score axes, accuracy per tier, and accuracy by family on the hard tier and sealed set. Further out is better on every spoke.

  • A: CLM-8B — system-one-open · Score 8.6 (#80)
  • B: Jev 1.13.0 — Jev (TypeSafe, closed) · Score 63.3 (#4)

The four score axes

Radar: the four score axes, two systemsThe four score axes, CLM-8B vs Jev 1.13.0. Intelligence: 22.4 vs 53.1; Calibration: 39.8 vs 76.3; Speed: 93.6 vs 83.3; Cost: 78.4 vs 52.0.50100Intelligence22.4 · 53.1Calibration39.8 · 76.3Speed93.6 · 83.3Cost78.4 · 52.0
0–100, the values in the table. A label-only system has no calibration (counted as 0).

Accuracy per tier, incl. sealed

Radar: accuracy per tier, incl. sealed, two systemsAccuracy per tier, incl. sealed, CLM-8B vs Jev 1.13.0. Easy: 67% vs 100%; Standard: 39% vs 99%; Judge: 71% vs 95%; Hard: 36% vs 74%; Sealed: 24% vs 37%.50100Easy67% · 100%Standard39% · 99%Judge71% · 95%Hard36% · 74%Sealed24% · 37%
Share correct per tier; Sealed = the 308 private decisions, aggregate only.

Current question set by family (hard + sealed)

Radar: current question set by family (hard + sealed), two systemsCurrent question set by family (hard + sealed), CLM-8B vs Jev 1.13.0. Ambiguous / abstain: 31% vs 43%; Judge: 32% vs 54%; Long policy: 23% vs 44%; Multi-hop: 22% vs 64%; Probability: 44% vs 63%; Temporal / numeric: 22% vs 28%; Trade-off: 18% vs 55%; Routing: 50% vs 100%; Trap / adversarial: 38% vs 83%; Paraphrase: 43% vs 64%; Safety judge: 38% vs 38%.50100Ambiguous /abstain31% · 43%Judge32% · 54%Long policy23% · 44%Multi-hop22% · 64%Probability44% · 63%Temporal /numeric22% · 28%Trade-off18% · 55%Routing50% · 100%Trap /adversarial38% · 83%Paraphrase43% · 64%Safety judge38% · 38%
Share correct per family across the 220 hard-tier decisions (public and held out) and the 308 sealed decisions of v1.4, pooled; Routing is hard-tier only, Paraphrase and Safety judge sealed only.

Sealed set by family

Radar: sealed set by family, two systemsSealed set by family, CLM-8B vs Jev 1.13.0. Ambiguous / abstain: 32% vs 30%; Judge: 22% vs 34%; Long policy: 5% vs 28%; Multi-hop: 18% vs 45%; Paraphrase: 43% vs 64%; Probability: 50% vs 50%; Safety judge: 38% vs 38%; Temporal / numeric: 18% vs 29%; Trade-off: 19% vs 38%; Trap / adversarial: 25% vs 42%.50100Ambiguous /abstain32% · 30%Judge22% · 34%Long policy5% · 28%Multi-hop18% · 45%Paraphrase43% · 64%Probability50% · 50%Safety judge38% · 38%Temporal /numeric18% · 29%Trade-off19% · 38%Trap /adversarial25% · 42%
Share correct within each sealed family — system-level aggregates; the items stay private.

Availability and evidence

Openness
Code and weights marked open in the published row
License note
Apache-2.0 (code and CLM-v0.1-8B head); Qwen3-8B encoder Apache-2.0
Cost evidence
estimated; the board’s row disclosure contains the published basis.
Endpoint condition
our evaluator-owned Lium GPU pod (RTX PRO 6000), offline read-only container, author's server on loopback
Note on this row
Contrastive-LM/CLM commit cca045ffdb07b3ebcfe6938537cdeac5e14899c9, head Contrastive-LM/CLM-v0.1-8B rev 87655cb835bd76fd66c2da78e1e3709f7fa11a94 (clm-latest), Qwen3-8B rev b968826d9c46dd6066d109eabc6255188de91218 last-token pooling via vLLM. Authors' documented system_one path: state + instructions as state text, each option description as a candidate action, softmax over contrastive scores at temperature 1.0. Offline local open-weight inference on the frozen 308-item v1.4 set in a network-disabled, read-only container on an evaluator-owned Lium RTX PRO 6000 pod; no operator endpoint; no golds were exposed. Latency is in-process Engine.answer time on the serial standard+judge items with the self-hosted adjustment. Cost is estimated at the Qwen3-Embedding-8B hosted list price ($0.01/M input, same-size 8B pooling encoder) over CLM's measured encoder tokens; it is not a GPU bill.
Published source
https://github.com/Contrastive-LM/CLM

From the public v1.4.2.2 aggregate. Scores and ranks can change when a new release is published.

Read the full board and published method. The overall score is a composite, not raw accuracy.