LUMIERE

Artificial intelligence program that turns your photos or text into video.

We introduce Lumiere, a text-to-video propagation model designed to synthesize videos depicting realistic, diverse, and coherent motion, a crucial challenge in video synthesis.

To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once with a single pass in the model.

This is in contrast to existing video models that synthesize remote keyframes and then temporal super-resolution; This is an approach that inherently makes global temporal consistency difficult to achieve. Our model learns to directly render a full-frame-rate, low-resolution video by distributing both spatial and (importantly) temporal downsampling and upsampling and leveraging a pre-trained text-to-image propagation model. multiple space-time scales. We demonstrate state-of-the-art text-to-video conversion results and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized rendering.