Peek Behind the Sneaks: Project #SceneStitch Unlocks Creative-Aware Fill

Brian demonstrates #SceneStitch on stage at Adobe MAX 2017.

Reality has a way of messing up the perfect shot — the pristine landscape, except for that car; the famous landmark, except for that tourist. Most professional photographers and designers have become adept at removing unwanted objects from photos out of necessity.

It’s a need and a skill as old as photography itself, but the challenge is always the same — removing objects from a photo leaves a hole in the image that must be replaced.

Today, tools like the clone stamp or content-aware fill in Photoshop make it easy to replace portions of an image by taking another section of the photo and replicating it. These techniques work great for replacing smaller portions of an image, but start to look repetitive or unnatural when applied to large sections of a photo.

That’s a problem Brian Price, a senior research scientist at Adobe, hopes to solve with #SceneStitch, a sneak technology that leverages the power of artificial intelligence to seamlessly replace large sections of an image.

“Unlike content-aware fill, which uses pixels from elsewhere in the same photo, #SceneStitch uses Adobe Sensei to search millions of images in Adobe Stock and finds appropriate content for filling the image. It automatically pulls the best matches and provides a series of options,” Brian says.

In the past the only way to do that has been manually, and it’s typically a time-consuming challenge to find replacement imagery that matches the original’s subject, color, shading, and texture for a natural result.

By contrast, #SceneStich works by leveraging advancements in deep learning and computer vision to understand the content of an image and generate short descriptions of the overall content. Using this semantic understanding of the image, Adobe Sensei can search across the millions of images in Adobe Stock to find candidate images with matching scenes and content appropriate for filling the image.

Once the candidate images are selected, #SceneStitch identifies the appropriate objects or parts of the candidate images for filling the hole. “You wouldn’t want to grab a chunk of sky and stick it in the middle of a road,” Brian explains. “Once we’ve found the image, there’s also some compositing work that takes place to help match color and brightness for a natural look.”

Finally, #SceneStitch provides multiple replacement options and examples so the photo editor can select the best match. The result is a powerful, new capability for creatively cutting and replacing large portions of a photograph.

An example of creative foreground replacement using #SceneStitch.

Although Brian has a deep background in computer graphics and AI, it’s the practical and creative uses of his research that motivate him. “Growing up I always loved to take photos, but never really had any classes. I’m just a normal person who likes photos,” Brian says. “So the thing that excites me the most about #SceneStitch is the potential for creativity. It gives you more than just a way to fill the hole in a photo. It gives you an easy new way to rethink entire scenes by mixing and matching imagery across millions of possibilities.”

In the future, for example, it might be possible to specifically target the filling image. Instead of finding matching imagery, you could ask the computer to replace it with something specific or unique to your creative vision.

“I’m excited to see where we can push it. Twenty years ago, we’d only have been able to use a few hundred images and it would have taken computers weeks to solve these sorts of learning and vision challenges,” Brian says “Now we’re using millions of images, and we can do it in an instant. So instead of it being an academic curiosity, it’s something that can help real people make creative decisions. It enables new possibilities. It becomes something my kids can use to draw pictures, to explore their creativity, and create higher quality things.”

Key contributors to #SceneStitch include Brian Price, senior research scientist; Scott Cohen, principal scientist; Zhe Zhu, intern and student at Tsinghua University; and Mingyang Ling, research software developer (Adobe Research).

This story is part of a series that will give you a closer look at the people and technology that were showcased as part of Adobe Sneaks. Read other Peek Behind the Sneaks stories here.