Meta unveils new AI model that generates video with sound

Meta has built a new artificial intelligence model called Movie Gen that creates photorealistic video and audio clips based on user prompts, and claims it rivals tools from leading media generation startups such as OpenAI and ElevenLabs.Samples of Movie Gen productions provided by Meta show videos of animals swimming and surfing, as well as videos using real photos of people describing their actions such as painting on a canvas. and videos that use real photos of people to depict actions such as them painting on a canvas.

Movie Gen also generates background music and sound effects that are synchronized with the content of the video, and the tool can be used to edit existing videos, Meta said in a blog post. In one such video, Meta had the tool insert pom-poms into the hands of a man running alone in the desert, while in another it transformed a parking lot where a man was skateboarding from dry ground to ground covered in splash puddles.

Movie Gen produces up to 16 seconds of video and up to 45 seconds of audio. Blind tests shared by the company show that the model outperforms products from startups like Runway, OpenAI, ElevenLabs, and Kling.

It's unlikely that Meta will release Movie Gen for open use by developers, as it did with the Llama series of large language models, and says the company considers the risks of each model individually.Meta is working directly with the entertainment industry and other content creators on uses for Movie Gen, and will incorporate it into Meta's own products sometime next year Meta According to the blog post and a research paper Meta released on the tool, the company used a mix of licensed and publicly available datasets in building Movie Gen.

Translated with DeepL.com (free version)