#ProjectFastMask Brings New Speed and Accuracy to Object Selection in Video

Joon-Young Lee, a research scientist at Adobe, demonstrates #ProjectFastMask on stage at Adobe MAX 2018.

Object selection is the foundation of creative digital editing. Before you can apply an effect to a part of on image—from making a color change, adding a special effect or combining different elements together—it all starts with selecting the specific parts of the image you want to apply the filter or effect to. That’s why a significant portion of the tool palette in software like Adobe Photoshop and After Effects are dedicated to making and refining selections, creating layers or applying masks.

It’s a skill that can be difficult to master. That’s especially true for video, where a selection or mask needs to be adjusted for every frame of video, typically 24 or 30 frames for every second of footage. And, if the selection needs to capture an object or background that contains movement, maintaining accurate and visually pleasing selection boundaries quickly becomes a time consuming, labor-intensive task.

When Joon-Young Lee—a research scientist at Adobe—tried to edit his first video, he knew there had to be a better way. “I was able to learn Photoshop and become proficient pretty quickly, but when I tried to apply those same skills to creatively edit a video, I was surprised by how time intensive separating a foreground object from the background of a video sequence could be,” he says. “Although there are existing tools for propagating a selection, I found that they could be unreliable—especially if the object you were tracking was moving fast, passed behind other objects or occlusions, or complex backgrounds. I wanted to invent a better way to do it.”

The result is #ProjectFastMask, a sneak technology powered by Adobe Sensei. On stage at Adobe MAX 2018, Joon-Young demonstrated how #ProjectFastMask allows video editors to create an accurate selection mask for complex objects in motion, with as few as four-simple clicks. Better yet, that selection is maintained even with quick movement, backgrounds in motion or when the selected object goes out of view behind other objects in the scene.

A selection of a dancer is automatically propagated for each frame using #ProjectFastMask

With quick and accurate selections, Joon-Young was able to select an image of a person dancing down a street in Brooklyn, separate the dancer from the background, apply a color filter to the background layer, and insert special text effects in between the layers in just a matter of seconds. In another example he was able to track a selection of a kitten as it moved behind and then back out from a fence post with no noticeable degradation of the selection.

“We’re building on several years’ worth of work that uses AI and machine learning techniques to propagate selection in video,” says Joon-Young. “#ProjectFastMask uses a specific kind of neural network called a memory network. Essentially, we track the data about the selected object from previous frames, embed it into the memory network and use it to reconstruct the selection for each new (query) frame. It provides much more accurate results with the order of magnitude faster runtime than previous best techniques on public benchmarks.”

In the future, Joon-Young hopes to apply similar AI technology to solve new creative challenges. “We see great potential to apply memory networks toward other pixel-level estimation problems, including object tracking, video interactive segmentation, and inpainting,” he adds.

This story is part of a series that will give you a closer look at the people and technology that were showcased as part of Adobe Sneaks. Read other Peek Behind the Sneaks stories here.