A recent wave of video generation models has emerged, showcasing stunning picturesque quality. However, one of the current challenges in video generation is the ability to produce coherent large motions. Many leading models either generate small motion or exhibit noticeable artifacts when producing larger motions. To address this, we introduce VideoPoet, a large language model (LLM) capable of various video generation tasks such as text-to-video, image-to-video, video stylization, video inpainting and outpainting, and video-to-audio. Unlike alternative models, VideoPoet seamlessly integrates multiple video generation capabilities within a single LLM.
The diagram below illustrates the capabilities of VideoPoet. It can animate input images to produce motion and edit videos for inpainting or outpainting. For stylization, the model takes in a video with depth and optical flow information and adds contents on top to create text-guided style.
Using language models (LLMs) for training has the advantage of reusing scalable efficiency improvements from existing LLM training infrastructure. However, LLMs operate on discrete tokens, which can pose challenges in video generation. To overcome this, VideoPoet uses video and audio tokenizers to encode video and audio clips as sequences of discrete tokens. These tokens can be converted back into their original representation. VideoPoet trains an autoregressive language model across different modalities using multiple tokenizers.
Below are some examples generated by VideoPoet:
VideoPoet can generate longer videos by conditioning on the last 1 second of video and predicting the next 1 second. The model can extend the video while preserving the appearance of objects. It also allows interactive editing of generated video clips. Additionally, VideoPoet can animate input images and control camera movements based on text prompts.
VideoPoet was evaluated on text-to-video generation using various benchmarks. The model’s performance was compared to other approaches, and user preferences were collected. VideoPoet achieved high ratings in terms of text fidelity and motion interestingness.
