Improve model card: add pipeline tag, paper link, and structured information
Browse filesThis PR improves the model card for LiveTalk. Specifically, it:
- Adds the `image-to-video` pipeline tag to the metadata to improve discoverability.
- Links the original paper and the official GitHub repository.
- Includes key highlights of the model such as its real-time performance and multimodal capabilities.
- Adds a sample usage section with the inference command provided in the GitHub documentation.
- Adds the BibTeX citation for researchers.
README.md
CHANGED
|
@@ -2,9 +2,9 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
tags:
|
| 4 |
- agent
|
|
|
|
| 5 |
---
|
| 6 |
|
| 7 |
-
|
| 8 |
# LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
|
| 9 |
|
| 10 |
<div align="center">
|
|
@@ -21,12 +21,54 @@ tags:
|
|
| 21 |
</a>
|
| 22 |
</p>
|
| 23 |
|
| 24 |
-
This is the model for https://github.com/GAIR-NLP/LiveTalk
|
| 25 |
|
| 26 |
-
**LiveTalk** enables real-time multimodal interactive avatar video generation through an improved on-policy distillation approach.
|
| 27 |
-
By distilling bidirectional diffusion models into causal, few-step autoregressive models, LiveTalk achieves over **20× speedup**,
|
| 28 |
-
enabling seamless real-time interactive experience.
|
| 29 |
|
| 30 |
<p align="center">
|
| 31 |
<img src="assets/overview.png" alt="LiveTalk System Overview" width="100%">
|
| 32 |
-
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
tags:
|
| 4 |
- agent
|
| 5 |
+
pipeline_tag: image-to-video
|
| 6 |
---
|
| 7 |
|
|
|
|
| 8 |
# LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
|
| 9 |
|
| 10 |
<div align="center">
|
|
|
|
| 21 |
</a>
|
| 22 |
</p>
|
| 23 |
|
| 24 |
+
This is the official model repository for [LiveTalk](https://github.com/GAIR-NLP/LiveTalk).
|
| 25 |
|
| 26 |
+
**LiveTalk** enables real-time multimodal interactive avatar video generation through an improved on-policy distillation approach. By distilling bidirectional diffusion models into causal, few-step autoregressive models, LiveTalk achieves over **20× speedup**, enabling a seamless real-time interactive experience.
|
|
|
|
|
|
|
| 27 |
|
| 28 |
<p align="center">
|
| 29 |
<img src="assets/overview.png" alt="LiveTalk System Overview" width="100%">
|
| 30 |
+
</p>
|
| 31 |
+
|
| 32 |
+
## ⭐ Highlights
|
| 33 |
+
|
| 34 |
+
- **Real-Time Generation**: Achieves 24.82 FPS throughput with 0.33s first-frame latency.
|
| 35 |
+
- **Multimodal Conditioning**: Supports text, image, and audio inputs for flexible avatar control.
|
| 36 |
+
- **Efficient Inference**: Reduces inference time from ~83s to real-time through 4-step diffusion distillation.
|
| 37 |
+
- **Multi-Turn Coherence**: Demonstrates competitive performance against state-of-the-art models in multi-round interaction benchmarks.
|
| 38 |
+
- **End-to-End System**: Provides integration with audio language models for conversational AI applications.
|
| 39 |
+
|
| 40 |
+
## 🚀 Get started
|
| 41 |
+
|
| 42 |
+
### Installation
|
| 43 |
+
|
| 44 |
+
For detailed setup instructions, including environment configuration and dependency installation, please refer to the [official GitHub repository](https://github.com/GAIR-NLP/LiveTalk).
|
| 45 |
+
|
| 46 |
+
### Inference
|
| 47 |
+
|
| 48 |
+
Once the environment and checkpoints are prepared, you can execute the inference script:
|
| 49 |
+
|
| 50 |
+
```bash
|
| 51 |
+
bash ./scripts/inference.sh
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
**Input Requirements:**
|
| 55 |
+
- **Image**: Reference image of the person (JPG/PNG format).
|
| 56 |
+
- **Audio**: Speech audio file (WAV format, 16kHz sample rate recommended).
|
| 57 |
+
- **Text Prompt**: Description of the desired video characteristics.
|
| 58 |
+
|
| 59 |
+
## 🔍 Method Overview
|
| 60 |
+
|
| 61 |
+
LiveTalk addresses challenges in distilling multimodal video diffusion models by using an improved on-policy distillation recipe. It introduces curated multimodal conditions, converged ODE initialization, and aggressive optimization to eliminate training instability (like flickering or black frames) while delivering high-quality, lip-synced results.
|
| 62 |
+
|
| 63 |
+
## 📚 Citation
|
| 64 |
+
|
| 65 |
+
If you find this work useful for your research, please cite:
|
| 66 |
+
|
| 67 |
+
```bibtex
|
| 68 |
+
@article{livetalk2025,
|
| 69 |
+
title={LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation},
|
| 70 |
+
author={Chern, Ethan and Hu, Zhulin and Tang, Bohao and Su, Jiadi and Chern, Steffi and Deng, Zhijie and Liu, Pengfei},
|
| 71 |
+
journal={arXiv preprint arXiv:2512.23576},
|
| 72 |
+
year={2025}
|
| 73 |
+
}
|
| 74 |
+
```
|