Transitive Representation Learning Enhances Histopathology Annotation

We introduce SpatialWhisperer: zero-shot cell-type annotation of H&E images, learned from molecular measurements of single cells.

Moritz Schaefer, Zoe Piran, Nils Philipp Walter, Animesh Awasthi, Christoph Bock, Jure Leskovec, Zinaida Good

International Conference on Machine Learning (ICML), 2026 · Seoul, Korea

Motivation

Cancer diagnosis depends on identifying cells in tissue images, which today still takes time-consuming assays or expert annotation. We built SpatialWhisperer, a model that annotates cell types in H&E images zero-shot, learned from abundant molecular measurements of single cells.

Question. How can we predict cell types in H&E despite the lack of fine-grained annotations?

Zero-shot lung-cancer annotation — Zero-shot annotation of a held-out lung-cancer H&E section from free-text queries, matching the expert ground truth.

Idea: Transfer transcriptomics annotations to histopathology

Rich single-cell identity annotations are abundant in the transcriptomics domain, but absent for histopathology images. We transfer this knowledge to H&E by composing two paired data sources: spatial transcriptomics pairs images (I) with gene expression (G); single-cell atlases pair gene expression (G) with text (T) annotations. Gene expression is the shared bridge that transitively links images to text.

Trimodal transitive bridge schematic — Disjoint paired datasets covering *image ↔ gene expression* (HEST-1K) and *gene expression ↔ text* (CellWhisperer). A shared gene-expression bridge transitively induces the *image ↔ text* alignment.

Method: trimodal contrastive learning

SpatialWhisperer aligns the three modalities in one shared embedding space. Training uses 921K I↔G pairs from HEST-1K and 1.08M G↔T pairs from CellWhisperer; the shared G modality transitively induces I↔T, which is never directly trained. A single InfoNCE objective combines the two losses, L = L_IG + L_GT. Encoders are UNI2 (image), Geneformer (expression) and BioBERT (text); the UNI2 and Geneformer backbones stay frozen, while the BioBERT text encoder and all projection heads are trained.

Results: SOTA on histopathology single-cell annotation

SpatialWhisperer zero-shot annotates cell types in H&E image patches, outperforming all five tested baselines in mean AUROC across three independent benchmarks.

3-benchmark AUROC barplot — Mean AUROC across three benchmarks (PathoCell: 13 cell types, colorectal; Lizard: 3, colon; PanNuke: 4, 19 organs) for SpatialWhisperer (mean over 3 seeds) and five baselines.

Why it works: a bound that tightens

To explain why an alignment we never train on (I↔T) should emerge, we relate its loss to the quality of the two observed pairs. Encoder quality is captured by the two margins we care about in contrastive learning: matched pairs are close (similarity ≥ 1−ε) and mismatched pairs stay far apart (≤ η).

Our finding: the unobserved I↔T InfoNCE loss is provably upper-bounded by (ε, η) and tightens as the observed pairs align (ε, η ↓).

Lemma. Under these margins, the unobserved per-sample InfoNCE loss is bounded, ℓ_I→T ≤ log(1 + N·e^r(ε,η)/τ), and tightens as ε, η ↓.

Characterizing transitive representation learning

1. The value of task-matched data. Transitive data helps most when task-specific labels are scarce (low-N).

Subsampling experiment — Macro-AUROC vs. task-matched data fraction across three modality-pair benchmarks; the trimodal model (red) beats the bimodal baseline (blue), most at low data.

2. Bridge-modality overlap. Imperfect overlap in the shared bridge adds only O(√δ) slack (deviation δ), so transfer degrades gracefully. Harmonizing annotation style (curation) raises image ↔ text similarity and improves typing.

Curation effect on image-text similarity — Per-sample image ↔ text similarity: LLM-*curated* QUILT-1M labels align better than the originals (Mann–Whitney p < 10⁻¹⁸⁷).

Takeaways: Cell-level interpretation of routine H&E

SpatialWhisperer transfers the rich cell-type knowledge of single-cell transcriptomics onto ordinary histopathology images, zero-shot and without task-specific labels, turning cheap and ubiquitous H&E into a molecularly interpretable, cell-resolved readout.

Outlook. Applying SpatialWhisperer to emerging single-cell-resolution spatial data is a natural next step, which we expect to sharpen alignment and further boost performance.

Resources

Paper (OpenReview): openreview.net/forum?id=Ze7U293Zw4
Code: github.com/zinagoodlab/spatialwhisperer
Weights: huggingface.co/Good-Lab/spatialwhisperer

Citation

@inproceedings{schaefer2026transitive,
  title={Transitive Representation Learning Enhances Histopathology Annotation},
  author={Schaefer, Moritz and Piran, Zoe and Walter, Nils Philipp
          and Awasthi, Animesh and Bock, Christoph
          and Leskovec, Jure and Good, Zinaida},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning},
  series={Proceedings of Machine Learning Research},
  publisher={PMLR},
  year={2026}
}