How We Calibrate ADMET-AI for Tox Pre-Screening

What Powers the Tox Panel

ToxScreen's public hERG and CYP panel runs on ADMET-AI, an open-source message-passing graph neural network (built on the chemprop architecture) trained to predict absorption, distribution, metabolism, excretion, and toxicity endpoints directly from molecular structure (Swanson et al., ADMET-AI, Bioinformatics 2024). We do not run any structure-prediction, docking, or physics-based binding model on the tox panel: inference is a deterministic, CPU-based forward pass through a trained neural network, taking roughly 1–7 seconds per compound.

Around that base model we add two layers that turn a raw model output into a decision-grade prediction:

• Isotonic probability calibration — maps the model's raw output onto a calibrated probability, so that a reported "70% likely inhibitor" actually corresponds to a 70% empirical frequency.
• Tanimoto applicability-domain (AD) abstention — measures how similar a query compound is to the model's training distribution and abstains when the compound falls outside that domain. The model is allowed to say "I don't know" rather than guess on chemistry it has never seen.

RDKit physicochemical descriptors (molecular weight, LogP, TPSA, PAINS alerts) sit in front of all of this as a pre-flight sanity layer only — they catch malformed or obviously problematic inputs but do not produce the toxicity call.

Why Calibration Matters

A raw neural-network output is just a number between 0 and 1, and an uncalibrated number is dangerous: a model can be highly discriminative yet systematically over- or under-confident. Isotonic regression fits a monotonic mapping from raw outputs to calibrated probabilities using held-out reference data, so that the probabilities ToxScreen reports are usable as probabilities — for ranking, for setting a triage threshold, and for honest communication of risk.

Calibration does not, by itself, fix a model that is wrong on unfamiliar chemistry. That is the job of the applicability-domain layer, and it is the single feature we consider ToxScreen's strongest differentiator.

Applicability-Domain Abstention: Knowing When We Don't Know

Most ADMET tools, including widely used free ones, return a number for every input regardless of whether the model has ever seen anything like it. ToxScreen instead computes Tanimoto similarity between the query compound and the model's training set. When a compound is too dissimilar — outside the model's applicability domain — ToxScreen flags low confidence and abstains rather than emitting a number that looks authoritative but is unsupported. This is the difference between a tool that is occasionally confidently wrong and one that tells you exactly where its predictions can be trusted.

The Real Numbers

Below are the actual calibration AUROC values our engine produces, measured on a random split with three seeds and roughly 1,800 compounds per target. We publish them as they are — including where hERG falls short — because honest reporting is the point.

CYP Inhibition Is Strong

The three CYP endpoints — CYP3A4, CYP2D6, and CYP2C9 — are where calibrated ML ADMET is genuinely excellent. At AUROC 0.94–0.95 on a random split with n ≈ 1,800 per target, the model cleanly separates inhibitors from non-inhibitors. These are the endpoints where a computational pre-screen most reliably triages out clear liabilities before you commit to in-vitro work.

CYP3A4 metabolises roughly half of marketed drugs, and inhibition drives clinically significant drug–drug interactions, particularly for narrow-therapeutic-index drugs. CYP2D6 is polymorphic, so inhibiting it in a poor-metaboliser phenotype can produce dangerously elevated exposure. CYP2C9 metabolises NSAIDs, sulfonylureas, and S-warfarin. On this last point a clarification we keep front-and-centre: warfarin is a CYP2C9 substrate, not an inhibitor — a distinction we corrected after an earlier review and one worth stating plainly, because conflating the two is a common source of error in tox annotation.

hERG Is the Hard One — and We Say So

The human ether-à-go-go related gene (hERG) potassium channel is the single most important anti-target in drug discovery: inhibition causes QT prolongation, which can degenerate into torsades de pointes. It is also the hardest endpoint in our panel. On a random split, hERG comes in at AUROC 0.69 — a FAIL by our internal gate, and we report it that way rather than dressing it up.

There is a real nuance underneath that number. ADMET-AI alone reaches roughly 0.71 AUROC on de-leaked novel chemistry, and random-split benchmarks tend to understate performance on the out-of-distribution compounds that matter most in early discovery. With ToxScreen's calibration layer (isotonic re-weighting + ChemBERTa applicability domain), that rises to ~0.84 AUROC on the same novel-chemistry test set. So the honest framing is: CYP inhibition is strong (AUROC 0.94–0.95); hERG is harder — 0.69 on a random split, ~0.71 ADMET-AI baseline and ~0.84 after calibration on novel chemistry — so we calibrate hERG conservatively and lean hard on applicability-domain abstention, flagging out-of-domain compounds instead of guessing.

We do not claim hERG is our best-calibrated target. It is the one that most needs the AD-abstention safety net, and that is exactly why we built one.

Determinism and Reproducibility

Because inference is a single deterministic forward pass (no sampling, no random seeds at scoring time), re-running the same SMILES against the same model and calibration configuration returns an identical calibrated probability up to floating-point precision. Every ToxScreen report records the engine, the calibration configuration, and a timestamp, so a result can always be traced back to the exact configuration that produced it.

The Bottom Line

ToxScreen's tox panel is calibrated, honest ML ADMET: ADMET-AI as the base model, isotonic calibration for usable probabilities, and Tanimoto applicability-domain abstention so the tool tells you when your chemistry is too novel to trust. CYP inhibition is strong; hERG is the genuinely hard endpoint, and we report the failing random-split number alongside the novel-chemistry result instead of hiding it. For the full pipeline, targets, and known limitations, see the Methodology page.

Published by the EstimaBio / ToxScreen team. Questions or corrections? Email info@toxscreen.ai.