score_signal 2026-11-01
Eval results for score_signal with question set 2026-11-01. Every response records the question set that answered it in question_set_version, so you can match a decision you received to these numbers.
Labeled set
Section titled “Labeled set”synthetic-score_signal(synthetic): Generated bypackages/engine/evals/scripts/generate-synthetic.tswith seed 20261101: 240 rows of invented firm events against one invented ICP, each with a label and model answers drawn around that label with noise. No row comes from a real firm, person or reply, so the numbers measure whether the pipeline (input builder, injection guard, calibration, threshold) stays consistent from one change to the next. They are not a measure of real-world accuracy.
The design partner’s labeled sets are not yet included for this decision.
Results
Section titled “Results”| Metric | This run | Baseline | Change (points) |
|---|---|---|---|
| Precision (calibrated, default threshold) | 0.762 | 0.762 | 0.0 |
| Recall (calibrated, default threshold) | 0.762 | 0.762 | 0.0 |
| F1 | 0.762 | 0.762 | 0.0 |
| Default threshold | 0.225 | ||
Raw precision (model p at 0.5) |
0.750 | ||
Raw recall (model p at 0.5) |
0.857 | ||
| Raw F1 | 0.800 | ||
| Labeled rows | 240 | ||
| Rows the injection guard fired on | 0 |
Gate: passed. CI runs the eval on every pull request. The build fails if precision or recall at the default threshold drops more than two points (0.02) below the committed baseline (packages/engine/evals/baseline.json). F1 and the raw columns are reported, not gated. A change to a question set ships only with an eval run.
How this was run
Section titled “How this was run”Precision and recall are measured at the default threshold: calibrated p from a global calibration map, cross-fitted over two folds so no row is scored by a map fitted on itself, with the threshold chosen for F1 on the other fold. F1 is computed from those precision and recall values. The raw columns apply the uncalibrated model probability at 0.5, so the gap shows what calibration changes. During the eval the model answers are the answers recorded in each labeled row, served by the fixture adapter (model version jev-fixture). No call is made to Jev, the production decision model.
| Field | Value |
|---|---|
| Adapter | fixture |
| Model version | jev-fixture |
| Run date | 2026-11-08 |
| Commit | 4ddf9174f1c8 |
| Clean tree | yes |