A Peek Behind the Scenes: #CAFVIDEO — Content Aware Fill for Video

Video editors and visual effects artists who use Content Aware Fill in Photoshop — which can convincingly erase an object from an image with a single keystroke — have been eagerly awaiting its debut in an Adobe video application. With the recent sneak peek of Content Aware Fill in After Effects at Adobe MAX 2018, we’ve seen the research once called Project Cloak move closer to reality: soon it will become easier than ever to cleanly remove elements from video, saving hours that used to be spent painstakingly erasing or replacing objects, sometimes frame by frame.

Going from algorithm to product feature is a long process, but we’re excited about where these tools are headed, and we know you are too: we’ve heard from hundreds of users with a wide variety of backgrounds, all hoping to get their hands on “Cloak” as soon as possible.

One of the most obvious applications is to conceal production equipment that’s gotten into a shot, like boom microphones or special effects wires. This is especially critical for immersive VR projects, as there’s nowhere “off-camera” to hide crew, tripods or lights. There are also creative possibilities: creating clean video plates for compositing or removing visual distractions like a car driving through the background of a scene or dust on a lens. Visual effects artists already need to do all of these tasks on a regular basis, and time saved building the foundation of a composite is time that can be spent polishing it to a higher standard.

Technology can be exciting — and a little scary

Not only is Content Aware Fill for video groundbreaking in its implementation — just like the original Content Aware Fill in Photoshop — it can raise questions about responsible usage. That’s why we’re previewing it now: we want you to be a part of that conversation. As Mark Randall, VP for Creativity, wrote in an earlier blog post:

It’s actually the entire point of showcasing Sneaks technology at MAX and other conferences throughout the year. Sneaks are always experimental in nature and show off some of the latest technology being developed in our labs. We share new features and capabilities, often well before they make their way into products, so that we can get real, immediate feedback from our customers and community about what those innovations mean to them — what’s useful, what can be better, what can be more impactful and meaningful to creative people in their daily lives.

Sometimes that generates controversy, but it also generates opportunity. We’re getting feedback about what makes Adobe technology useful, as well as what concerns people, and we can factor that into our future plans as well as our ongoing conversations with customers and advisory boards that shape our development.

Image manipulation is not new. It has a long history that predates the digital revolution. But digital tools have democratized methods of altering an image’s content after the image has been captured, sometimes dramatically. As these technologies advance, those changes become more seamless and more difficult for a viewer to distinguish from reality. These tools also become easier for everyday people to use, not just professionals with advanced training. And this is a key challenge of incorporating technology into our daily lives: it can change our expectations of what’s possible.

At Adobe, we think technology should be used to advance creativity and drive efficiency for creators. And we believe that Content Aware Fill for video can do just that. Of course, new technologies always have the potential to be used for nefarious purposes. While it’s impossible for us to monitor intent, we develop all our tools thoughtfully and purposefully, supported by complementary technological advances that help us document and trace the digital manipulation of photos or videos. Embedded metadata, forensic analysis, watermarking and other tools can all help determine ownership or authenticity.

Technology can be defined by you

The After Effects team plans to release Content Aware Fill in an upcoming release of After Effects. But before we can ship, we need your feedback. We would like to offer a select group of users an early preview of what we’ve done so far, so you can tell us what we need to do to make sure Content Aware Fill is all you expect it to be.

To apply, go to adobeprerelease.com, click on Available Programs, find After Effects Content-Aware Fill, and click Apply.