Commit ·
55707ba
1
Parent(s): ed9ee75
Improve model card: add paper/code links, license and sample usage (#1)
Browse files- Improve model card: add paper/code links, license and sample usage (e185044424134682546c6ea82f462242035fd129)
Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>
README.md
CHANGED
|
@@ -1,11 +1,52 @@
|
|
| 1 |
---
|
| 2 |
-
license: other
|
| 3 |
library_name: pytorch
|
|
|
|
| 4 |
pipeline_tag: audio-to-audio
|
| 5 |
---
|
| 6 |
|
| 7 |
# HoliTok
|
| 8 |
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
|
|
|
| 2 |
library_name: pytorch
|
| 3 |
+
license: apache-2.0
|
| 4 |
pipeline_tag: audio-to-audio
|
| 5 |
---
|
| 6 |
|
| 7 |
# HoliTok
|
| 8 |
|
| 9 |
+
HoliTok is a continuous holistic speech tokenization model designed for unified generation-understanding modeling. It encodes 48 kHz speech into a compact 25 Hz sequence of 128-dimensional latents. It is designed to preserve signal-level fidelity, incorporate semantic information, and maintain strong latent learnability.
|
| 10 |
+
|
| 11 |
+
- **Paper:** [HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding](https://huggingface.co/papers/2605.29948)
|
| 12 |
+
- **Code:** [GitHub Repository](https://github.com/bovod-sjtu/HoliTok)
|
| 13 |
+
|
| 14 |
+
This repository hosts runtime-only HoliTok checkpoints for inference. The checkpoints are loaded by the HoliTok inference package through the public aliases `HoliTok-Base` and `HoliTok-Unite`.
|
| 15 |
+
|
| 16 |
+
## Sample Usage
|
| 17 |
+
|
| 18 |
+
You can use the HoliTok Python API for audio tokenization, reconstruction, and semantic feature extraction.
|
| 19 |
+
|
| 20 |
+
```python
|
| 21 |
+
import torch
|
| 22 |
+
from holitok import HoliTok, SemanticModule
|
| 23 |
+
|
| 24 |
+
model = HoliTok.from_pretrained("HoliTok-Unite", device="cuda:0")
|
| 25 |
+
audio = torch.randn(1, 1, 48000, device="cuda:0")
|
| 26 |
+
|
| 27 |
+
# [B, 2 * latent_dim, T], concat(mu, log_std)
|
| 28 |
+
posterior = model.encode_posterior(audio)
|
| 29 |
+
|
| 30 |
+
# [B, latent_dim, T]
|
| 31 |
+
latents = model.posterior_mean(posterior)
|
| 32 |
+
|
| 33 |
+
# [B, 1, samples]
|
| 34 |
+
recon = model.decode(latents)
|
| 35 |
+
|
| 36 |
+
semantic = SemanticModule.from_pretrained("HoliTok-Unite", device="cuda:0")
|
| 37 |
+
features = semantic(latents.transpose(1, 2)) # [B, T, 1536]
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
## Citation
|
| 41 |
+
|
| 42 |
+
```bibtex
|
| 43 |
+
@misc{li2026holitokacoutinuousholistictokenization,
|
| 44 |
+
title={HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding},
|
| 45 |
+
author={Bohan Li and Shi Lian and Hankun Wang and Yiwei Guo and Yu Xi and Zhihan Li and Da Zheng and Colin Zhang and Kai Yu},
|
| 46 |
+
year={2026},
|
| 47 |
+
eprint={2605.29948},
|
| 48 |
+
archivePrefix={arXiv},
|
| 49 |
+
primaryClass={cs.SD},
|
| 50 |
+
url={https://arxiv.org/abs/2605.29948},
|
| 51 |
+
}
|
| 52 |
+
```
|