:
Fine-Controllable and Expressive Freestyle Portrait Animation

Yue Ma1* Hongyu Liu1* Hongfa Wang2,3* Heng Pan2* Yingqing He1
Junkun Yuan2 Ailing Zeng2 Chengfei Cai2 Heung-Yeung Shum1,3 Wei Liu2✝ Qifeng Chen1✝
*Equal Contribution. Corresponding Author.
1HKUST 2Tencent, Hunyuan 3Tsinghua University

arxiv[Paper]      github[Billbill]      twitter[Twitter]      YouTube[YouTube]      google[Try it!]     

Demo Video!

Abstract

We present Follow-Your-Emoji, a diffusion-based framework for portrait animation, which animates a reference portrait with target landmark sequences. The main challenge of portrait animation is to preserve the identity of the reference portrait and transfer the target expression to this portrait while maintaining temporal consistency and fidelity.

To address these challenges, Follow-Your-Emoji equipped the powerful Stable Diffusion model with two well-designed technologies. Specifically, we first adopt a new explicit motion signal, namely expression-aware landmark, to guide the animation process. We discover this landmark can not only ensure the accurate motion alignment between the reference portrait and target motion during inference but also increase the ability to portray exaggerated expressions (i.e., large pupil movements) and avoid identity leakage. Then, we propose a facial fine-grained loss to improve the model's ability of subtle expression perception and reference portrait appearance reconstruction by using both expression and facial masks. Accordingly, our method demonstrates significant performance in controlling the expression of freestyle portraits, including real humans, cartoons, sculptures, and even animals. By leveraging a simple and effective progressive generation strategy, we extend our model to stable long-term animation, thus increasing its potential application value. To address the lack of a benchmark for this field, we introduce \textbf{EmojiBench}, a comprehensive benchmark comprising diverse portrait images, driving videos, and landmarks. We show extensive evaluations on EmojiBench to verify the superiority of Follow-Your-Emoji.

Gallery

Single Motion + Multiple Portraits

BibTex

@article{ma2024followyouremoji,
title={Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation},
author={Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, Qifeng Chen},
year={2024},
eprint={2406.01900},
archivePrefix={arXiv},
primaryClass={cs.CV}
}