“Firefly for Video” will generate sound effects and music, edit footage via text.
On Monday, Adobe announced plans to add AI video synthesis tools to Premiere Pro and After Effects, reports Reuters. This follows an announcement last week that AI-powered text-based video editing will come to Premiere Pro in May.
Adobe’s generative video effects will come courtesy of Adobe Firefly. Firefly, an image synthesis mode we previously covered in March, will expand to include video and audio generation tools. Premiere Pro is Adobe’s flagship video editing application, a popular industry tool for assembling TV shows and Hollywood films.
Currently, performing certain special effects or editing tasks can often be labor-intensive. Adobe aims to streamline that process with generative AI assistance. In a demo reel posted on its blog, Adobe showed “Firefly for Video” generating a background soundtrack, generating sound effects, changing the visual season in a video, creating a visual effect on the word “molten,” locating and brightening a person’s face, and generating audio transcripts via text commands.
In its promotional blog, Adobe mentions features we might expect from its new AI tools, including some listed above (like generating sound effects and music), but also text-based color enhancements, text and logo animations, automatic storyboard creation, and creative “co-pilots” that can supposedly generate how-to’s to help users master editing skills.
If these kinds of text-based video editing tools seem familiar, it’s because Runway teased similar capabilities in September, although Runway has yet to deliver on those promises with the fidelity previewed in its teaser. Similarly, it’s probably wise to take Adobe’s marketing video with a grain of salt until the product is released.
(And only time will tell if Adobe Firefly will ever be able to generate a video of Will Smith eating spaghetti that can rival the masterpiece devised by ModelScope, an open source video synthesis model.)
In a separate but related announcement last week, Adobe announced it is integrating text-based video editing into Adobe Premiere Pro, which will allow editors to trim and re-arrange video based on automatically detected transcripts of speech pulled from video clips. Here’s how Adobe describes the feature:
Source : https://arstechnica.com/information-technology/2023/04/adobe-teases-generative-ai-video-tools/