Bringing generative AI to video with Adobe Firefly Video Model

Animated Gif of a woman putting on glasses.

We first launched Adobe Firefly in March 2023, and since then we’ve delivered rapid innovation with new models in imaging, design and vectors. These Firefly models have quickly grown to power some of the most popular features across Creative Cloud and Express like Generative Fill in Photoshop, Generative Remove in Lightroom, Generative Shape Fill in Illustrator and Text-to-Template in Express. Along the way, we’ve received incredible feedback from the creative community and enterprise customers alike — and in total our community has generated over 12B images and vectors making Firefly and the features it powers some of the fastest adopted by our community.

We all know that video is the currency of engagement today — and we’re excited to share a peek at the upcoming Firefly Video Model and some of the revolutionary professional workflows it’ll power in our industry-leading video tools like Premiere Pro, available starting in beta later this year.

Over the past several months, we’ve worked closely with the video editing community to advance the Firefly Video Model. Guided by their feedback and built with creators’ rights in mind, we’re developing new workflows leveraging the model to help editors ideate and explore their creative vision, fill gaps in their timeline and add new elements to existing footage.

Just like our other Firefly generative AI models, editors can create with confidence knowing the Adobe Firefly Video Model is designed to be commercially safe and is only trained on content we have permission to use — never on Adobe users’ content.

We’re excited to share some of the incredible progress with you today — all of which is designed to be commercially safe and available in beta later this year. To be the first to hear the latest updates and get access, sign up for the waitlist here.

Get notified when Firefly Video is available.

Join the waitlist

A new era of video editing

The ever-increasing demand for fresh, short-form video content means editors, filmmakers and content creators are being asked to do more and in less time. Today not only do editors cut picture, but they’re also tasked with color correction, titling, visual effects, animation, audio mixing and more. At Adobe, we’re leveraging the power of AI to help editors expand their creative toolset so they can work in these other disciplines, delivering high-quality results on the timelines their clients require.

Common editorial tasks — like navigating gaps in footage, removing unwanted objects from a scene, smoothing jump cut transitions, and searching for the perfect b-roll takes time. When performed well, addressing these tasks can make the difference between a compelling and emotional narrative, rather than one that distracts from the story you’re trying to tell.

Sometimes, sharing creative intent with your team and the stakeholders that green light and fund the work can also be a challenge, requiring many rounds of communication. At Adobe, we provide tools such as Frame.io to streamline teamwork by enabling a uniquely integrated review and approval process. And now, we are delivering AI tools to help facilitate an even better creative process that helps take the tedium out of post-production. Our AI tools give editors more time to explore new creative ideas, the part of the job they love, while also setting them up for successful collaboration with the larger team. Not only does Adobe facilitate streamlined work process, but now, with Generative AI, we’ve made it even easier and faster for editors and motion designers to create their best work in record time. Watch below to see how we’re taking video editing to new heights using Firefly.

With Firefly Text-to-Video, you can use text prompts, a wide variety of camera controls, and reference images to generate B-Roll that seamlessly fills gaps in your timeline.

Image of woman wearing glasses.

All generations shown below were created with Adobe’s Firefly Video Model and were generated in under 2 minutes.

https://main--blog--adobecom.hlx.page/en/assets/video/media_1090eebf441cd466e882455b0c7afe198af87c1a5.mp4#autoplay

Prompt: Cinematic closeup and detailed portrait of a reindeer in a snowy forest at sunset. The lighting is cinematic and gorgeous and soft and sun-kissed, with golden backlight and dreamy bokeh and lens flares. The color grade is cinematic and magical.

https://main--blog--adobecom.hlx.page/en/assets/video/media_1f195afae3f2040088b84c9828219388a15c391b8.mp4#autoplay

Prompt: Slow-motion fiery volcanic landscape, with lava spewing out of craters. the camera flies through the lava and lava splatters onto the lens. The lighting is cinematic and moody. The color grade is cinematic, dramatic, and high-contrast.

https://main--blog--adobecom.hlx.page/en/assets/video/media_1f500f2db3eb239757a53cb3fd37e5da4688ae813.mp4#autoplay

Prompt: Miniature adorable monsters made out of wool and felt, dancing with each other, 3d render, octane, soft lighting, dreamy bokeh, cinematic.

The Firefly Video Model excels at generating videos of the natural world. When production misses a key establishing shot needed to set the scene, generate an insert with camera motion, like landscapes, plants or animals.

https://main--blog--adobecom.hlx.page/en/assets/video/media_1bbba64a5bd69abc277a1ba918acced2563921335.mp4#autoplay

Prompt: Footage of a camera on a drone flying over a desert with wind blowing over the dunes creating waves in the sand below.

Need more complementary shots? Fill the gap in your timeline with a generated clip based on a reference frame.

Original footage:

Generated clip:

https://main--blog--adobecom.hlx.page/en/assets/video/media_1aa1fceac992b3db71ba6673c1706ef178c3ffd31.mp4#autoplay

Prompt: Detailed extremely macro closeup view of a white dandelion viewed through a large red magnifying glass

Combined sequence:

With the Firefly Video Model you can leverage rich camera controls, like angle, motion and zoom to create the perfect perspective on your generated video.

Screenshot of Firefly Video Model in use.

https://main--blog--adobecom.hlx.page/en/assets/video/media_14f97af70c25061a10988ede7d7c610b2c9dc5424.mp4#autoplay

Prompt: Drone shot going between the trees of a snowy forest at sunset golden hour. The lighting is cinematic and gorgeous and soft and sun-kissed, with golden backlight and dreamy bokeh and lens flares. The color grade is cinematic and magical.

The more detailed your prompts are, the better the model can leverage that depth to generate inspirational imagery and b-roll.

https://main--blog--adobecom.hlx.page/en/assets/video/media_11b4d12d5e6681df54b0cefd90f48be9b4ac4f52a.mp4#autoplay

Prompt: Cinematic closeup and detailed portrait of a man in the middle of a rainy street. the lighting is moody and dramatic. The color grade is blues and teals. the man is extremely realistic with detailed skin texture and stubble on his face. movement is subtle and soft. the camera doesn’t move. heavy film grain and textures. Beads of water slowly rolling down his face.

The Firefly Video Model supports a broad variety of use cases including creating atmospheric elements like fire, smoke, dust particles and water against a black or green background that can then be layered over existing content using blend modes or keying inside Adobe’s tools like Premiere Pro and After Effects.

Original media:

Flame overlay:

Combined sequence:

Ideate 2D and 3D animation, including Claymation, that can be shared with creative collaborators to communicate intent.

https://main--blog--adobecom.hlx.page/en/assets/video/media_1ae9ec3a75440a55e68a1d8916985dc818b348a49.mp4#autoplay

Prompt: Stop motion 2d animation made of felt of an egg cooking in a frying pan in a sunlit beautiful kitchen on the stovetop.

https://main--blog--adobecom.hlx.page/en/assets/video/media_1ca85d8415516f84198c8db3f6066170dbd43d3d3.mp4#autoplay

Prompt: Hand-drawn simple line art, a young kid looking up into space with a wondrous expression on his face.

https://main--blog--adobecom.hlx.page/en/assets/video/media_1481bdae71b94c534282791ea0514fb231325be11.mp4#autoplay

Prompt: An adorable kawaii cheeseball on the moon smiling, 3d render, octane, soft lighting, dreamy bokeh, light sparkles floating in the background.

Brainstorm ideas for custom text effects to share with clients for feedback.

https://main--blog--adobecom.hlx.page/en/assets/video/media_112518a0ed00afc37bc6e24243a7c19f0cb98c840.mp4#autoplay

Prompt: Macro detailed shot of water splashing and freezing to spell the word "ICE”.

With Image-to-Video, you can bring an existing still shot or illustration to life by transforming them into stunning live action clips.

Reference image:

Galaxy far away in outer space.

https://main--blog--adobecom.hlx.page/en/assets/video/media_190d5eee53cb631a094203b0fae1fff31b1dd1a96.mp4#autoplay

Prompt: Zoom out of the galaxy to reveal that it is contained inside the pupil of a human eye, detailed macroscape.

And coming later this year to Premiere Pro (beta), Generative Extend allows you to extend clips to cover gaps in footage, smooth out transitions, or hold on shots longer for perfectly timed edits.

Watch how the editor uses Generative Extend powered by the Adobe Firefly Video Model in Premiere Pro to hold on a shot longer to match the crescendo of the audio.

And here you can see the final output with the generated frames creating the perfect edit.

We’re excited by all the recent advancements on the Adobe Firefly Video Model and look forward to continuing to partner with the community to build generative AI into the Adobe tools and workflows you rely on.

Get notified when Firefly Video is available.

Join the waitlist