#ProjectSmoothOperator Brings Intelligent, Automated Cropping to Video Publishing

Nicolas Huynh Thien demos #ProjectSmoothOperator at Adobe MAX 2018.

Video is a critical tool for engaging audiences on social media. It’s also a huge challenge. On platforms like Facebook, Twitter and Instagram, competing content is generated every second and attention spans are fleeting. That makes it especially important to optimize video for each platform and device.

Traditionally, preparing video for each platform has required an editor to create unique versions of each video — something that can quickly become a tedious, time-consuming and expensive process. Video shot for a cinematic, 16×9 aspect ratio often has action or visuals that stretch across the entire screen, so the footage can’t simply be centered and cropped for a 1:1 square preview window, a 4×3 aspect ratio for traditional monitors or a vertical window for mobile devices.

#ProjectSmoothOperator, an Adobe Sneak technology showcased at MAX 2018, seeks to automate that process. It provides both marketers and editors an easier way to optimize and post cross-platform video, giving new life to existing video assets without requiring manual, tedious version editing.

“SmoothOperator started as a conversation between my group (Sensei Applied Science) and a product manager for Adobe Premiere Rush,” explains Nicolas Huynh Thien, an applied scientist and machine learning engineer at Adobe. “There was strong demand for a feature to publish direct from the app to multiple platforms all at once, rather than manually upload to YouTube, Facebook, Instagram and others, and we wanted to do it without requiring a lot of extra editing work at publishing time. We started collaborating with two researchers who were tackling a similar problem before the project was selected to be presented at MAX.”

Project SmoothOperator uses technology powered by Adobe Sensei to examine a video and generate a heat map representing the saliency of the subject and action for each frame. Then, given the heat map and desired aspect ratio of the output, it calculates a global camera path, shifting left and right, or up and down across the frame to capture all the relevant action. “We had to treat it as an optimization problem. You don’t want the motion of the camera to be too extreme, so it’s about finding the right balance between static shots, camera pans and smooth transitions between the two. You don’t want to miss the action — but you don’t want too much camera motion either, because it can be disorienting,” Nicolas explains.

At Adobe MAX, Nicolas showed the audience several examples of SmoothOperator in action, generating multiple aspect ratio edits of video with subjects as diverse as an aerial shot of a car driving across the desert, a freestyle skier soaring down a mountain and even a shot of a dog catching a frisbee, accurately generating camera motions that captured the core action of each shot.

For Nicolas, it’s gratifying to work on a project that boosts creativity and removes the hurdles of tedious tasks for marketers and editors. “Our team joined Adobe earlier this year as part of their acquisition of Uru, a video intelligence and computer vision start-up, and it’s been really rewarding to be able to tackle a project like this and have an immediate visible impact on a stage like MAX . I’m looking forward to seeing how SmoothOperator evolves — we’re already working on speeding up processing times, examining various ways to handle on-screen text overlays and exploring future product possibilities for this technology,” he says.

This story is part of a series that will give you a closer look at the people and technology that were showcased as part of Adobe Sneaks. Read other Peek Behind the Sneaks stories here.