A Starring Role for Immersive Technology

Adobe special effects and computer graphics expert, Bhautik Joshi is helping craft innovative storytelling technology.

Image source: Starlight Studios LLC & Kristen Stewart

Hollywood dreams can spring from many sources. There’s passion for acting and emotional performance that fuels the flame of stardom. Driven to shape narratives and tell stories, others seek the creative freedom of being directors and writers. But there’s a third group that works with equal inspiration behind the scenes as they advance technology, making it possible to tell stories in magical new ways.

For Bhautik Joshi, senior research engineer with Adobe Research, Hollywood dreaming started with film and special effects when he was just seven-years-old, watching the peerless sci-fi adventure movie — Star Wars.

“From the moment I saw that Imperial Cruiser fill the screen in front of me, flying over my head, seat shaking and lasers firing, I knew I wanted to be a part of bringing new realities and imagination to life,” he says.

Bhautik Joshi, senior research engineer with Adobe Research.

It was that single moment of movie magic that sparked his interest in storytelling, special effects, and computer graphics. It drove him to study computer science and biomedical engineering at the University of New South Wales. Ultimately, it led to a career path that’s included medical simulation, game development, and eventually the company that inspired it all — Industrial Light & Magic.

“I was at Industrial Light & Magic for six years,” he says. “I worked on animation, pre-visualization work, motion capture, and stereo. It gave me an amazing grounding in real-time graphics.”

Later, Bhautik spent time at Flickr where he utilized machine learning and AI to analyze their catalog of 12 billion images, identify outstanding photos, and then deliver the most inspirational highlights to photographers.

He sees it all as a career path in storytelling technology that naturally led him to joining Adobe. “My whole life I’ve been building tools for artists, whether that’s gaming, film, or photography, and that’s Adobe’s thing. It’s a lot of what I liked about being at ILM, but with Adobe I can have an impact at a much larger scale,” he explains. “What I like best about being here is really being able to get to the heart of the challenges storytellers face when dealing with immersive content.”

That desire to make a big impact, along with his interest in the intersection of storytelling and technology, inspired Bhautik to experiment with a technology called neural style transfer. This AI –based technique makes it possible to redraw one image using the style of another and can be applied to film and photo editing.

This compelling capability caught the attention of film star and director, Kristen Stewart. She needed a way to create a dreamy, surreal visual style for her short film, Come Swim. Bhautik and Kristen collaborated to create a visual style reminiscent of impressionist artwork in support of key story elements. They eventually published an academic paper together, Bringing Impressionism to Life with Neural Style Transfer in Come Swim, generating quite a bit of attention from both the Hollywood and AI communities.

Come Swim uses Neural Style Transfer. Left: content image, middle: style image, right: upsampled result. Image source: Starlight Studios LLC & Kristen Stewart.

It’s a great example of what’s possible when new, emerging technologies like AI and VR are made more accessible to storytellers. “I’ve been in and out of VR for a long time now, the technology hasn’t always kept pace with our vision. What’s different about it this time is that we have 2.6 billion smartphones out there that can double as immersive viewing devices. I want to give artists the ability to make use of that canvas,” he explains.

To that end, Bhautik is working with the Adobe Premiere team, and directly with artists in Hollywood, to develop new capabilities in 360 video — from camera capture, to editing workflows, and even the publishing platforms for immersive content.

“A key issue with VR is that there’s not enough people using it. We need to make 360 cameras more accessible to the semi-pro and consumer audiences. We need to make sure that what the camera captures ends up displayed correctly when it gets edited and published. We can make it a lot easier to display 360 images correctly so that the viewer never gets lost or disoriented when they’re ‘inside’ the image,” he explains.

It’s a huge challenge requiring camera makers, software developers, and publishers to work together, but Bhautik believes it will happen so long as the community is responsive to the needs of storytellers. “VR is changing and evolving all the time,” he adds. “We don’t want to put the brakes on, we just want to put some shape around it. My hope is that artists will build on that and push it in new directions.”