Peek Behind the Sneaks: #ProjectCloak — Filmmakers Gain the Power of Invisibility

Geoff Oxholm demonstrates #ProjectCloak on stage at Adobe MAX 2017.

Sometimes the camera is too good at capturing reality, whether that’s dust on the lens, the tourist that wandered into frame, or the unsightly billboard marring a pristine landscape. To get the perfect shot, filmmakers need the power of invisibility—the ability to make unwanted objects disappear.

It’s an ability that photographers have long had to themselves. With technologies like content-aware fill, even amatuers can remove unsightly objects from a photo while preserving the natural look of an image.

Removing objects from video, on the other hand, has traditionally required painstaking, professional skill. Frame-by-frame corrections have to be made—that’s 24-30 images for every second of film. And even with tools like content-aware fill, the finished product can result in ghost artifacts or other blemishes.

Enter #ProjectCloak, a sneak technology with the potential to provide an easy-to-use cloak of invisibility for filmmakers. Geoff Oxholm, a research engineer at Adobe, demonstrated the technology at MAX 2017. He explains, “This has been an active area of research for years. A principal scientist in our group, Eli Shechtman, wrote his 2004 PhD thesis on it. Recently there have been some great observations that we’re relying on. The secret sauce is in the careful use of dense tracking and content-aware-fill.”

#ProjectCloak works by allowing the user to select an area of the video they want to hide. The software then estimates the motion of the corresponding scene, searching forward and backward in the video to replace missing pixels with the background as seen in other frames.

For videos where the region behind the object is never seen, the system employs Adobe’s content-aware-fill to guess what is missing. Ultimately, the algorithm figures out how to replace the unwanted pixels with new ones that best match the appearance of the scene.

The result is easy-to-use invisibility, without the ghosting and artifacts that result from manual, frame-by-frame corrections.

#ProjectCloak makes it easy to remove unwanted objects from video, such as this lamppost, even when the camera is moving.

It’s a magical combination of technologies. And exactly the kind of work Geoff likes to do. He’s been coding and finding practical applications for software since his first programmable calculator in high school. After pursuing a degree in software engineering, he formed a start-up with some friends creating software for NASA to evaluate the sleep patterns of astronauts. Then he earned a Master’s degree and Ph.D. , where his work focused on developing 3D reconstructions out of two-dimensional photographs. Later he joined Adobe, where one of his first projects was to take video clips and turn them into perfect, seamlessly repeating loops . “Building things has always been a delightful experience for me,” he says. “Seeing other people benefit from them is icing on the cake.”

Today, Geoff is excited by the future potential for #ProjectCloak. “It’s not just about removing objects, there are all sorts of artifacts an editor might want to remove from the shot, like the shadow of a drone, or the tripod at the base of a 360 camera,” he says. “The cool thing about #ProjectCloak is that it works well even if the camera is moving. But there are still some limitations. Really complex shots, combined with depth changes, can result in some parallax artifacts. Or if the background behind the object you’re removing is also moving—such as the cascading water of a fountain—it will have a hard time with that. Our next challenge is to tackle some of those problems, and make the technology even more useful.”

#ProjectCloak was a collaboration between Geoff; Oliver Wang, senior research sceintist; Eli Shechtman, principal scientist; Mike Lukáč, research scientist; and Xin Wang, Adobe intern and student at UCSB. Want to dig a little deeper into the technology powering #ProjectCloak? Check out the story from Adobe Research.

This story is part of a series that will give you a closer look at the people and technology that were showcased as part of Adobe Sneaks. Read other Peek Behind the Sneaks stories here.