Finding 7 · 2026-08-16
D3: the powered holdout cleared 12/12 on both targets, and 2YXJ’s FAIL flipped to PASS
A properly-sized blinded holdout (n=12 per target, scaffold-distinct from calibration, RDKit formula-verified, 3 genuine replicates) replaced an earlier n=3 test. 7AAD stayed PASS; 2YXJ’s earlier FAIL turned out to be a single unlucky compound at n=3, and cleared 12/12 once properly powered.
What we expected
A properly-sized blinded holdout (n=12 per target, scaffold-distinct from calibration, RDKit formula-verified, 3 genuine replicates) should give a more reliable readout than the earlier n=3 test.
What happened
| target | old verdict | new verdict | bp range | at frozen threshold | at the new constrained threshold |
|---|---|---|---|---|---|
| 7AAD | PASS 3/3 (noise) | PASS 12/12 | 0.933–0.996 | 12/12 @ 0.500 | 12/12 @ 0.897 |
| 2YXJ | FAIL 1/3 (noise) | PASS 12/12 | 0.684–0.963 | 12/12 @ 0.534 | 12/12 @ 0.542 |
2YXJ’s FAIL was noise, as suspected. At n=3 the 75% bar can only be met by 3/3; one unlucky compound failed the target. At n=12 it clears every compound. This is the third gate that week whose verdict inverted once it was properly powered.
Why it happened
The holdout compounds were, by design, the most potent compounds available — curate_blinded_holdout.py sorts by pChEMBL descending and takes the top 12 (7AAD: pChEMBL 9.70–10.70; 2YXJ: pChEMBL 8.85–9.22). The author flags this as a self-authored limitation: that reasoning was too comfortable.
What changed
The result is scoped honestly: what 12/12 demonstrates is that Boltz-2 correctly recognises sub-nanomolar binders as binders, not that it discriminates near threshold or ranks across the potency range. A harder holdout spanning pChEMBL 5–10, including compounds that should fail, is recorded as owed. The current result is to be quoted as “recognises strong binders,” not “validated hit-gate.”
“These holdouts are the most potent compounds available… and that reasoning was too comfortable.”
Bring us the number you are least sure about.
That is usually the one worth thirty minutes.