Magic Animate

magic-research / magic-animate

[CVPR 2024] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

Zhongcong Xu · Jianfeng Zhang · Jun Hao Liew · Hanshu Yan · Jia-Wei Liu · Chenxu Zhang · Jiashi Feng · Mike Zheng Shou


National University of Singapore   |   ByteDance

📢 News

• [2023.12.4] Release inference code and gradio demo. We are working to improve MagicAnimate, stay tuned!
• [2023.11.23] Release MagicAnimate paper and project page.

🏃‍♂️ Getting Started

Download the pretrained base models for StableDiffusion V1.5 and MSE-finetuned VAE.

Download our MagicAnimate checkpoints.

Please follow the huggingface download instructions to download the above models and checkpoints, git lfs is recommended.

Place the based models and checkpoints as follows:

magic-animate
|----pretrained_models
  |----MagicAnimate
    |----appearance_encoder
      |----diffusion_pytorch_model.safetensors
      |----config.json
    |----densepose_controlnet
      |----diffusion_pytorch_model.safetensors
      |----config.json
    |----temporal_attention
      |----temporal_attention.ckpt
  |----sd-vae-ft-mse
    |----config.json
    |----diffusion_pytorch_model.safetensors
  |----stable-diffusion-v1-5
    |----scheduler
       |----scheduler_config.json
    |----text_encoder
       |----config.json
       |----pytorch_model.bin
    |----tokenizer

…

View on GitHub

I tried Magic Animate with a super famous Japanese comedian’s photo on Google Colab T4(free tier), but the result wasn’t what I expected lol.

But still, I think Magic Animate is awesome.

Another one that is kind of funny.

The following is the Google Colab I used.

llm_on_GoogleColab/MagicAnimate_colab.ipynb at main · koji/llm_on_GoogleColab
Contribute to koji/llm_on_GoogleColab development by creating an account on GitHub.
github.com

Note

If you run the above Colab on T4, you will need to reduce sampling from 25 to around 15(12–15) because of T4’s memory size.

The entire process will take around 10 minutes.