---
license: other
license_name: penn-academic-software-license
license_link: LICENSE
pipeline_tag: image-segmentation
tags:
  - pathology
  - histopathology
  - medical-imaging
  - segmentation
  - whole-slide-image
  - interactive-segmentation
  - foundation-model
---

# VISTA-PATH

An interactive foundation model for pathology image segmentation and quantitative
analysis in computational pathology.

- 📄 Paper: https://www.arxiv.org/abs/2601.16451
- 💻 Code: https://github.com/zhihuanglab/VISTA-PATH
- 🔬 Integrated in [TissueLab](https://www.tissuelab.org/)

VISTA-PATH integrates visual context, textual class prompts and expert-guided
interaction. Pre-trained on over 1.6 million samples, it generalizes across
organs and tissue types, supports human-in-the-loop refinement, and enables
clinically interpretable analysis through survival-associated morphological
features.

## Architecture

| Component | Source | Role |
|---|---|---|
| Text encoder | `vinid/plip` (CLIP text tower) | Encodes the class prompt `"an image of {class_name}"` as a token sequence |
| Segmentation trunk | `facebook/mask2former-swin-small-ade-semantic` | Swin encoder + pixel decoder + masked-attention transformer decoder |
| Box prompt encoder | `facebook/sam-vit-base` (frozen `prompt_encoder`) | Encodes an optional bounding-box prompt as two corner tokens |

Twenty object queries cross-attend to the concatenated `[text tokens ; box tokens]`
sequence before the Mask2Former decoder, and a per-query binary head produces a
foreground probability map for the prompted class. When no box is given, two
learnable "no-box" tokens replace the SAM corner tokens, so the same checkpoint
serves both the prompt-free and the box-prompted settings.

Inference settings that must match the checkpoint: input resolution 512, 20 queries.

## Files

```
pytorch_model.bin     # released weights (~506 MB)
models/backbones.py   # CustomSegmentationModel
inference.py          # prompt-free / tissue-box whole-slide inference
inference_bbx.py      # box-prompted inference driven by a label mask
inference_utils.py    # model construction, checkpoint loading, sliding windows
utils.py              # visualization + segmentation metrics
idx_to_names/         # label index -> class name mappings
examples_BRCA/        # toy ROI image/mask pairs
```

## Usage

Download the repository and run either entrypoint. The three backbones above are
pulled from the Hub on first run.

```python
from huggingface_hub import snapshot_download
local = snapshot_download("zhihuanglab/VISTA-PATH")
```

Prompt-free segmentation of named classes over a slide:

```bash
python3 inference.py \
  --infer_vis_dir ./results \
  --checkpoint_file pytorch_model.bin \
  --image_file /path/to/slide.svs \
  --class_names "Tumor" "Stroma" \
  --crop_size 2048 --overlap 256 \
  --bbx_random 1
```

Box-prompted segmentation, where a label mask supplies one box per class present
in each window:

```bash
python3 inference_bbx.py \
  --infer_vis_dir ./results \
  --json_file ./idx_to_names/BRCA.json \
  --checkpoint_file pytorch_model.bin \
  --image_dir ./examples_BRCA/images \
  --mask_dir ./examples_BRCA/masks \
  --bbx_random 0
```

`--bbx_random` is the probability of dropping the box prompt: `1` runs prompt-free,
`0` conditions every window on its box. Only a `.jpg` overview is written by
default; add `--save_mask` for the full-resolution label mask and `--save_prob`
for per-class probability maps.

Environment: Python 3.12, torch 2.4.0+cu118 with torchvision 0.19.0+cu118,
transformers 4.46.1, openslide-python with `openslide-bin`. The cu118 build covers
`sm_50` through `sm_90` (V100 up to H200); Blackwell cards (RTX 50 series, B200)
need a cu128 build instead - torch 2.10.0+cu128 with torchvision 0.25.0+cu128.
See `environment.yml` and the installation section of the README in the GitHub repo.

## Citation

```bibtex
@misc{liang2026vistapath,
  title        = {VISTA-PATH: An interactive foundation model for pathology image segmentation and quantitative analysis in computational pathology},
  author       = {Liang, Peixian and Li, Songhao and Koga, Shunsuke and Li, Yutong and Alipour, Zahra and Tang, Yucheng and Xu, Daguang and Huang, Zhi},
  year         = {2026},
  eprint       = {2601.16451},
  archivePrefix= {arXiv},
  primaryClass = {cs.CV},
  doi          = {10.48550/arXiv.2601.16451},
}
```

## License

Released under the Penn Academic Software License Agreement (see `LICENSE`):
non-commercial academic research use only, with attribution. Commercial use,
sale, or other monetization of the model or its derivatives requires prior
approval.