Morphic Frames to Video

Introduction

The Morphic Frames to Video model is a LoRA designed for high-quality video generation by interpolating between a sequence of input image frames. Built upon the powerful Wan2.2 architecture, this model excels at creating smooth and coherent motion from static images.

Its key capabilities include generating seamless transitions between two distinct images and creating fluid animations from multiple keyframes. This makes it an ideal tool for animators, video editors, and creators looking to bring static art to life or produce dynamic visual effects.

The primary use cases demonstrated are:

• Two-Frame Transitions: Morphing one image into another, such as transforming a character into an object.
• Multi-Frame Interpolation: Animating a sequence of keyframes to produce a continuous video, like making a character stand up and wave.

How to Use

Follow these steps to set up and run the model.

Step 1: Set Up the Repository

First, clone the official Morphic repository from GitHub:

git clone https://github.com/morphicfilms/frames-to-video.git

For environment installation, it is recommended to follow the official Wan2.2 installation guide. Alternatively, you can run the provided setup script:

bash setup_env.sh

Step 2: Download Weights

You will need to download both the base Wan2.2 model and the Morphic Frames to Video LoRA weights.

1. Download Wan2.2 I2V Base Model:

   huggingface-cli download Wan-AI/Wan2.2-I2V-A14B --local-dir ./Wan2.2-I2V-A14B
   

2. Download Morphic LoRA Weights:

   huggingface-cli download morphic/Wan2.2-frames-to-video --local-dir ./morphic-frames-lora-weights
   

Step 3: Run Inference

The model can be run in two primary modes: two-frame transition or multi-frame interpolation.

For a two-frame transition:

torchrun --nproc_per_node=8 generate.py \
    --task i2v-A14B \
    --size 1280*720 \
    --frame_num 81 \
    --ckpt_dir ./Wan2.2-I2V-A14B-Interpolation \
    --high_noise_lora_weights_path ./morphic-frames-lora-weights/lora_interpolation_high_noise_final.safetensors \
    --dit_fsdp \
    --t5_fsdp \
    --ulysses_size 8 \
    --image examples/transition9_1.png \
    --prompt "A clown, slowly transforms into a poster." \
    --img_end examples/transition9_2.png

For multi-frame interpolation:

torchrun --nproc_per_node=8 generate.py \
    --task i2v-A14B \
    --size 1280*720 \
    --frame_num 81 \
    --ckpt_dir ./Wan2.2-I2V-A14B-Interpolation \
    --high_noise_lora_weights_path ./morphic-frames-lora-weights/lora_interpolation_high_noise_final.safetensors \
    --dit_fsdp \
    --t5_fsdp \
    --ulysses_size 8 \
    --image examples/pink_1.png \
    --prompt "The animated girl rises up from her chair and waves hi to the camera as the camer zooms in." \
    --img_end examples/pink_4.png \
    --middle_images examples/pink_2.png examples/pink_3.png \
    --middle_images_timestamps 0.4 0.7

Trigger Word

No trigger word is required to use this LoRA model.

Use Cases

This model is specialized for creating video sequences from a set of input frames. The provided documentation highlights two main use cases.

Creating Smooth Transitions

You can generate a video that seamlessly morphs a starting image into an ending image. This is perfect for creating artistic transformations, visual effects, or unique scene transitions. The example provided shows a clown slowly transforming into a poster, guided by a simple text prompt.

Animating Between Keyframes

The model can interpolate motion between multiple keyframes. By providing a start frame, an end frame, and one or more intermediate frames, you can create a fluid animation. This is useful for character animation or bringing storyboards to life, as shown in the example of an animated girl rising from a chair and waving.

Examples

The documentation provides two clear examples of the model's capabilities.

Example 1: Clown Transformation

• Type: Two-frame transition
• Input: A starting image of a clown and an ending image of a poster.
• Prompt: "A clown, slowly transforms into a poster."
• Output: A video depicting a smooth and continuous morphing effect where the clown's features and form gradually change into the poster.

Example 2: Animated Girl Sequence

• Type: Multi-frame interpolation
• Input: Four keyframe images of an animated girl in a sequence of actions.
• Prompt: "The animated girl rises up from her chair and waves hi to the camera as the camera zooms in."
• Output: A video that animates the girl's movement, filling in the frames between the provided images to create a fluid motion of her standing up and waving.

FAQ

Q: Do I need a trigger word to use this LoRA?

A: No, this model does not require a specific trigger word for activation.

Q: How do I specify intermediate frames for an animation?

A: Use the --middle_images argument to list the paths to your intermediate frames and the --middle_images_timestamps argument to define their position in the video's timeline (e.g., 0.4 means 40% of the way through). The number of images must equal the number of timestamps.

Q: What base model is this LoRA designed for?

A: This LoRA is built on and requires the Wan-AI/Wan2.2-I2V-A14B base model.

Q: Can I generate a video from just one image?

A: The provided documentation and examples focus on generating video between at least two images (a start and an end frame). It is designed for interpolation and transitions, not single-image animation.

Technical Details

• Base Model: Wan-AI/Wan2.2-I2V-A14B
• Model Type: wan-2.2-i2v LoRA
• File Format: The LoRA weights are provided in the .safetensors format.
• Training Details: The provided documentation does not include specific details about the training dataset or process.