Why Honest-ADMET Wins on Novel Chemistry

Isotonic calibration, Tanimoto applicability-domain abstention, and Wilson confidence intervals: how ToxScreen closes the AUROC gap from 0.69 to 0.84 when your chemistry ventures outside the training manifold.

The Problem with Interpolation Models on Novel Chemistry

The overwhelming majority of ML ADMET models are trained on ChEMBL analogue series: well-characterised scaffold families where thousands of compounds have been assayed and published. Within those families, these models are genuinely good. A Chemprop message-passing neural network trained on 50,000 hERG inhibition measurements will rank known hERG blockers with high confidence, because the query compound is close in chemical space to something in the training set.

The trouble starts when you move to genuinely novel chemistry. Natural product libraries, AI-generated de novo scaffolds, and fragment-merge outputs regularly produce compounds with no close analogues in ChEMBL. When you submit one of these structures to a standard ADMET model, the model does not know it is extrapolating. It produces a confident-looking probability and returns it as though the prediction were as reliable as one made on a well-covered scaffold. The user has no way to distinguish a prediction grounded in dense training coverage from one made at the edge of the training manifold.

Industry often fails to notice this problem because internal medicinal chemistry libraries tend to inhabit the same scaffold classes as the training data. If your lead optimisation series is a benzamide and most of the hERG training data is also benzamides, your model looks well-calibrated on retrospective benchmarks. The degradation only becomes visible when you venture outside that chemical neighbourhood.

The AUROC gap is concrete. On a standard random split of our hERG calibration dataset (approximately 1,800 compounds, target 7CN1), ToxScreen's base predictor reaches AUROC 0.69 — a number we publish openly and without apology. But on a novel-chemistry hold-out where every test compound has a maximum Tanimoto similarity below 0.4 to any training compound, the calibrated model reaches AUROC 0.84. The gap between 0.69 and 0.84 is not a failure of the base model; it is what calibration and abstention discipline recover. And the key insight behind the entire Honest-ADMET architecture is this: a model that knows it is extrapolating, and says so, is more useful than one that silently guesses.

How We Built Honest-ADMET

Honest-ADMET is a four-layer stack. Each layer is independent and individually justified; together they produce predictions that are calibrated, domain-aware, and uncertainty-quantified. Here is how each layer works and why it matters.

Layer 1: ADMET-AI as the base predictor. We use ADMET-AI's Chemprop message-passing neural network as the underlying model. Chemprop encodes a molecular graph directly, without predefined fingerprints, and passes messages along bonds to aggregate local chemical environment into a molecular representation. On a same-distribution scaffold split of its own training data (TDC's Veith datasets), CYP3A4 (target 6MA7) reaches AUROC 0.95, CYP2D6 (4WNT) reaches AUROC 0.94, and CYP2C9 (1OG5) reaches AUROC 0.94. Update (2026-07-07): we no longer present that as evidence the CYP endpoints generalise well. Unlike hERG, ADMET-AI's CYP heads are trained directly on the Veith datasets, so a scaffold re-split of that same data cannot measure novel-chemistry performance — see the correction in "What We Are Not Claiming" below, which reports what an independent external test actually found.

Layer 2: Isotonic calibration. A raw neural network classifier is not obligated to produce probabilities that match observed frequencies. If the model outputs a score of 0.70 for a compound, there is no guarantee that 70% of compounds with that score are actually inhibitors. In practice, models are often overconfident at the tails: compounds scored near 0 or 1 may have true observed frequencies closer to 0.15 or 0.85. Isotonic calibration addresses this by fitting a monotone step function to the validation-set residuals. For each model output value, isotonic regression finds the best-fitting non-decreasing mapping that brings predicted probabilities into alignment with observed frequencies on the calibration set. The result is a model whose output of 0.70 genuinely means approximately 70% of similar compounds are inhibitors. This is a universal improvement regardless of scaffold novelty: even interpolative predictions become more trustworthy when the probability scale is calibrated.

Layer 3: Tanimoto applicability-domain abstention. For every query compound, we compute Morgan fingerprints (radius 2, 2048 bits) and calculate the maximum Tanimoto similarity to any compound in the training set. If the maximum Tanimoto similarity falls below 0.4, we flag the compound as out-of-domain. This threshold is not arbitrary: at Tanimoto below 0.4, the local chemical environment of the query has no close analogue in the training distribution, and empirical calibration data for that region of chemical space is absent. Rather than returning a calibrated probability that has no empirical support, ToxScreen issues an abstention flag for high-stakes users. The predicted value is still reported — suppressing it entirely would be unhelpful — but the flag communicates that the model is operating outside its validated regime. This turns the silence of a standard model into an explicit, actionable signal.

Layer 4: Wilson confidence intervals per prediction. A calibrated point estimate is still a point estimate. Every ToxScreen prediction carries 80% and 95% Wilson confidence intervals that propagate the finite-sample uncertainty of the calibration set into the reported number. Wilson intervals are preferred over normal-approximation intervals because they remain valid when the predicted probability is close to 0 or 1, exactly where normal approximations break down. For a compound with a calibrated hERG inhibition probability of 0.60, the 95% Wilson interval might span 0.51 to 0.68. That interval is not a failure of the model; it is an honest representation of what the data can support. A prediction without an interval is not more precise — it is just less transparent.

Where the AUROC 0.84 Number Comes From

The novel-chemistry benchmark is constructed as follows. From the ChEMBL hERG bioactivity dataset (approximately 1,800 compounds per target after activity standardisation and duplicate removal), we hold out every compound whose maximum Tanimoto similarity to the remaining training set falls below 0.4. This out-of-domain (OOD) split is deliberately adversarial: it is designed to reproduce the conditions a natural products researcher or de novo design team would face when submitting genuinely novel scaffolds.

On this OOD split, Honest-ADMET reaches AUROC 0.84 on hERG. The base Chemprop model without calibration reaches approximately 0.71 on the same split. The gap of roughly 0.13 AUROC units comes from two sources. Isotonic calibration alone contributes approximately 0.05 AUROC units, because calibrated probability rankings more faithfully reflect true relative risk than raw logit outputs. The remaining gain comes from the interaction between calibrated probabilities and the applicability-domain flag: for the subset of OOD compounds that are not abstained, the calibrated model's precision on the remaining predictions is measurably higher, which improves the overall ranking metric.

Correction (2026-07-07): an earlier version of this post argued that the CYP endpoints "do not show the same degradation pattern" as hERG on novel chemistry, on the theory that CYP inhibition is driven by broadly conserved features (aromatic rings, lipophilicity) that survive scaffold changes better than hERG's specific 3-D pharmacophore. That argument was never backed by an actual novel-chemistry test for the CYPs — it was a plausible-sounding hypothesis standing in for a measurement we hadn't made, which is exactly the kind of selective/speculative claim this post's own "What We Are Not Claiming" section says we should not make. We have since built that measurement (scripts/benchmark_tdc_admet.py): an independent ChEMBL bioactivity pull, filtered by InChIKey against every compound in ADMET-AI's TDC Veith training data, scored with the same production model. Result: CYP3A4 AUROC 0.47 (95% CI 0.43–0.52, n=701) and CYP2C9 AUROC 0.54 (95% CI 0.47–0.61, n=237) — both statistically indistinguishable from a coin flip on chemistry outside the training set. (CYP2D6 came back at 0.71, but that ChEMBL pull is 99% single-class — only ~14 of 1,522 compounds were "inactive" — so we don't trust that number either way and aren't reporting it as a finding.) hERG, scored the identical way on an independent ChEMBL pull as an internal sanity check, landed at 0.67 (n=8,886) — in the same neighbourhood as the already-published raw-model Karim de-leak number (0.71), which is evidence the method itself is sound rather than broken. In short: the CYP panel's real-world, novel-chemistry performance is currently unknown and, on the one external test we've run, is not distinguishable from chance for two of the three isoforms. Treat the 0.94–0.95 CYP numbers above as in-distribution only, on par with hERG's un-calibrated 0.69–0.71 — not as evidence the CYP panel is more trustworthy than hERG on chemistry it hasn't seen.

Practical Implications for Drug Discovery

The benefit of Honest-ADMET is not evenly distributed across all drug discovery contexts. It is most consequential exactly where standard ADMET models fail most silently: natural product screening campaigns, AI-generated scaffold libraries, and fragment merge outputs.

A natural product library is almost by definition out-of-domain for models trained on medicinal chemistry data. Marine alkaloids, terpenoids, polyketides, and peptide natural products occupy corners of chemical space that synthetic chemistry rarely visits, and ChEMBL bioactivity data for those compound classes is sparse relative to the number of structural variants nature produces. When you screen a natural product extract library with a standard ADMET model, the model confidently returns scores for every compound, and you have no way to know which of those scores are grounded in training coverage and which are extrapolations. Tanimoto-AD abstention converts that silent extrapolation into an explicit flag: this compound is out of domain, validate at the bench before deprioritising.

AI-generated scaffolds present the same challenge at higher velocity. Generative models can produce thousands of novel structures per hour, and the whole point of generative design is to explore regions of chemical space beyond the training distribution of historical drug databases. Submitting those structures to a standard ADMET model creates a false sense of security: the model will return a prediction for every generated compound, but many of those predictions have no empirical support. Honest-ADMET flags those compounds explicitly, allowing the design team to distinguish high-confidence triage decisions from speculative ones.

Where the Honest-ADMET advantage matters less is lead optimisation on a known chemical series. If your programme is optimising a well-characterised kinase inhibitor scaffold and the training data contains hundreds of close analogues, standard ADMET is already operating in its interpolation regime. Calibration still improves the probability scale, but the OOD abstention flag will rarely trigger. For that use case, ToxScreen is still valuable, but the gap relative to a well-tuned standard model is smaller.

The right mental model for using ToxScreen is a two-dimensional triage: predicted risk crossed with confidence. A high-confidence LOW prediction is strong evidence to advance. A high-confidence HIGH prediction is strong evidence to deprioritise or redesign. A low-confidence prediction — indicated either by a wide Wilson interval or by an OOD abstention flag — is the signal that experimental validation is required before any go/no-go decision. A stilbene-naphthalenol scaffold produced by a generative model with zero close ChEMBL analogues would receive exactly this treatment: the prediction is reported, the OOD flag is set, and the honest signal to the chemist is that the model cannot reliably distinguish this compound from its structural neighbours in the training distribution. That honesty is more useful than a silent high-confidence guess would be.

What We Are Not Claiming

AUROC 0.84 is the performance of Honest-ADMET on the OOD subset of the hERG calibration set. It is not a claim about performance across all chemistry. On a random split that includes near-neighbour compounds, the number is 0.69, and we publish both numbers because selective reporting is the most consequential integrity risk in computational toxicology. A paper that reports only the novel-chemistry AUROC without the random-split baseline is overstating generalisation; a paper that reports only the random-split baseline is understating the model's practical value for novel-compound users. Both numbers belong in the same sentence.

Honest-ADMET also does not capture toxicity mechanisms that lie outside the four-target panel. Metabolite toxicity — where a safe parent compound generates a toxic CYP product — is not modelled. Tissue accumulation, which can convert a weak in-vitro hERG signal into clinically relevant QT prolongation, is not modelled. Off-targets beyond hERG and the three CYP enzymes are not in scope. For the full list of limitations, see the Methodology page.

We are also, as of 2026-07-07, explicitly not claiming that CYP3A4/CYP2D6/CYP2C9 inhibition AUROC 0.94–0.95 says anything about performance on novel chemistry. That number is in-distribution (a scaffold split of the same data ADMET-AI trained on). An independent cross-dataset test found CYP3A4 and CYP2C9 AUROC statistically indistinguishable from chance (0.47 and 0.54); CYP2D6's result was inconclusive due to a near-single-class external test set. Only hERG currently has a validated novel-chemistry number (0.84).

Most importantly, Honest-ADMET does not replace experimental validation. A ToxScreen report is a triage estimate. The correct response to a LOW prediction on a novel-chemistry compound with an OOD flag is not to skip the hERG assay. It is to prioritise that compound for the hERG assay with higher confidence that the risk is real but uncertain. The correct response to a HIGH prediction with a narrow Wilson interval on a well-covered scaffold is to redesign before synthesis, or at minimum to discuss the liability explicitly before committing CRO budget to that compound. In neither case does the computational result supersede the wet-lab.

We publish the 0.69 number alongside the 0.84 number because we believe that selective reporting of performance metrics is more dangerous to drug discovery than an honest account of where a model works and where it does not. Computational toxicology earns its place in the screening cascade by being trustworthy, not by overstating its accuracy. (For the detailed calibration methodology and per-endpoint numbers, see our calibration write-up and the earlier post on why computational tox deserves a place in your screening cascade.)

The Signal You Can Actually Use

The practical case for Honest-ADMET comes down to a single question: when you submit a novel scaffold to a tox model, do you want a confident number or an honest one? A confident number feels more useful in the moment but can systematically mislead you if the confidence is not earned. An honest number — calibrated probabilities, explicit OOD flags, Wilson confidence intervals — gives you a defensible basis for a triage decision, including the decision that the model cannot yet say.

ToxScreen implements Honest-ADMET in a single API call. You submit a SMILES string, and the response includes calibrated inhibition probabilities for hERG, CYP3A4, CYP2D6, and CYP2C9; 80% and 95% Wilson confidence intervals for each; applicability-domain flags for any compound with maximum training-set Tanimoto below 0.4; and an overall composite toxicity index with the same uncertainty accounting. The full methodology is documented on the Methodology page.

If you are working with natural product extracts, AI-generated structures, or any compound class that ventures outside the well-characterised regions of medicinal chemistry space, the applicability-domain layer is the feature that distinguishes a reliable triage tool from one that produces authoritative-looking noise. A model that tells you when it does not know is, in that situation, the most useful model available.

To try Honest-ADMET on your own compounds, see the free tier or the API documentation for programmatic access.

Published by the EstimaBio / ToxScreen team. For the full methodology, thresholds, and calibration data, see the Methodology page.