Image-to-Video
Safetensors
agent
nielsr HF Staff commited on
Commit
4efe1be
·
verified ·
1 Parent(s): 858dd03

Improve model card: add pipeline tag, paper link, and structured information

Browse files

This PR improves the model card for LiveTalk. Specifically, it:
- Adds the `image-to-video` pipeline tag to the metadata to improve discoverability.
- Links the original paper and the official GitHub repository.
- Includes key highlights of the model such as its real-time performance and multimodal capabilities.
- Adds a sample usage section with the inference command provided in the GitHub documentation.
- Adds the BibTeX citation for researchers.

Files changed (1) hide show
  1. README.md +48 -6
README.md CHANGED
@@ -2,9 +2,9 @@
2
  license: apache-2.0
3
  tags:
4
  - agent
 
5
  ---
6
 
7
-
8
  # LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
9
 
10
  <div align="center">
@@ -21,12 +21,54 @@ tags:
21
  </a>
22
  </p>
23
 
24
- This is the model for https://github.com/GAIR-NLP/LiveTalk
25
 
26
- **LiveTalk** enables real-time multimodal interactive avatar video generation through an improved on-policy distillation approach.
27
- By distilling bidirectional diffusion models into causal, few-step autoregressive models, LiveTalk achieves over **20× speedup**,
28
- enabling seamless real-time interactive experience.
29
 
30
  <p align="center">
31
  <img src="assets/overview.png" alt="LiveTalk System Overview" width="100%">
32
- </p>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: apache-2.0
3
  tags:
4
  - agent
5
+ pipeline_tag: image-to-video
6
  ---
7
 
 
8
  # LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
9
 
10
  <div align="center">
 
21
  </a>
22
  </p>
23
 
24
+ This is the official model repository for [LiveTalk](https://github.com/GAIR-NLP/LiveTalk).
25
 
26
+ **LiveTalk** enables real-time multimodal interactive avatar video generation through an improved on-policy distillation approach. By distilling bidirectional diffusion models into causal, few-step autoregressive models, LiveTalk achieves over **20× speedup**, enabling a seamless real-time interactive experience.
 
 
27
 
28
  <p align="center">
29
  <img src="assets/overview.png" alt="LiveTalk System Overview" width="100%">
30
+ </p>
31
+
32
+ ## ⭐ Highlights
33
+
34
+ - **Real-Time Generation**: Achieves 24.82 FPS throughput with 0.33s first-frame latency.
35
+ - **Multimodal Conditioning**: Supports text, image, and audio inputs for flexible avatar control.
36
+ - **Efficient Inference**: Reduces inference time from ~83s to real-time through 4-step diffusion distillation.
37
+ - **Multi-Turn Coherence**: Demonstrates competitive performance against state-of-the-art models in multi-round interaction benchmarks.
38
+ - **End-to-End System**: Provides integration with audio language models for conversational AI applications.
39
+
40
+ ## 🚀 Get started
41
+
42
+ ### Installation
43
+
44
+ For detailed setup instructions, including environment configuration and dependency installation, please refer to the [official GitHub repository](https://github.com/GAIR-NLP/LiveTalk).
45
+
46
+ ### Inference
47
+
48
+ Once the environment and checkpoints are prepared, you can execute the inference script:
49
+
50
+ ```bash
51
+ bash ./scripts/inference.sh
52
+ ```
53
+
54
+ **Input Requirements:**
55
+ - **Image**: Reference image of the person (JPG/PNG format).
56
+ - **Audio**: Speech audio file (WAV format, 16kHz sample rate recommended).
57
+ - **Text Prompt**: Description of the desired video characteristics.
58
+
59
+ ## 🔍 Method Overview
60
+
61
+ LiveTalk addresses challenges in distilling multimodal video diffusion models by using an improved on-policy distillation recipe. It introduces curated multimodal conditions, converged ODE initialization, and aggressive optimization to eliminate training instability (like flickering or black frames) while delivering high-quality, lip-synced results.
62
+
63
+ ## 📚 Citation
64
+
65
+ If you find this work useful for your research, please cite:
66
+
67
+ ```bibtex
68
+ @article{livetalk2025,
69
+ title={LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation},
70
+ author={Chern, Ethan and Hu, Zhulin and Tang, Bohao and Su, Jiadi and Chern, Steffi and Deng, Zhijie and Liu, Pengfei},
71
+ journal={arXiv preprint arXiv:2512.23576},
72
+ year={2025}
73
+ }
74
+ ```