Eval results

Precision, recall and F1 for each decision and question-set version, read from the result files committed in packages/engine/evals/results/. Each row says which labeled set it was measured on and where that set comes from.

Synthetic sets only

Every labeled set published here is synthetic. These numbers show the pipeline is consistent and that changes do not regress it. They do not show real-world accuracy. The design partner’s labeled sets are not included yet.

Results

Decision	Question set	Status	Labeled set	Provenance	Rows	Precision	Recall	F1	Change vs baseline (precision / recall, points)	Gate
qualify_firm	2026-11-01	current	synthetic-qualify_firm	Synthetic	240	1.000	0.900	0.947	0.0 / 0.0	passed
score_signal	2026-11-01	current	synthetic-score_signal	Synthetic	240	0.762	0.762	0.762	0.0 / 0.0	passed
pick_person	2026-11-01	current	synthetic-pick_person	Synthetic	240	0.903	0.949	0.926	0.0 / 0.0	passed
triage_reply	2026-11-01	current	synthetic-triage_reply	Synthetic	240	0.915	0.995	0.954	0.0 / 0.0	passed
right_person	2026-11-01	current	synthetic-right_person	Synthetic	240	0.768	0.843	0.804	0.0 / 0.0	passed
should_send	2026-11-01	current	synthetic-should_send	Synthetic	240	1.000	0.905	0.950	0.0 / 0.0	passed

How it is measured

Precision and recall are measured at the default threshold: calibrated p from a global calibration map, cross-fitted over two folds so no row is scored by a map fitted on itself, with the threshold chosen for F1 on the other fold. F1 is computed from those precision and recall values. The raw columns apply the uncalibrated model probability at 0.5, so the gap shows what calibration changes. During the eval the model answers are the answers recorded in each labeled row, served by the fixture adapter (model version jev-fixture). No call is made to Jev, the production decision model.

The gate

CI runs the eval on every pull request. The build fails if precision or recall at the default threshold drops more than two points (0.02) below the committed baseline (packages/engine/evals/baseline.json). F1 and the raw columns are reported, not gated. A change to a question set ships only with an eval run.

What does not run yet

• The design partner’s labeled sets are not included yet. When they are added, their results are published here per decision and question-set version, separately from the synthetic ones.
• No public labeled set is included yet.
• The live model is not evaluated. Because the answers are recorded, not produced by Jev, these numbers do not yet show how Jev itself scores on these rows, and they do not yet prove the raw Jev versus Verdict comparison on real data.
• Every v1 decision has at least one labeled set, so none is listed as not yet published.