Teenage Mutant Ninja Turtles Come to Life in VR using Adobe Character Animator at Comic-Con

Image courtesy of Nickelodeon.

To introduce its new Rise of the Teenage Mutant Ninja Turtles series, Nickelodeon couldn’t have asked for a better venue than Comic-Con 2018. The network used VR and Adobe Character Animator CC to immerse attendees in the turtle’s world, transporting them to a rooftop in New York City where they interacted live with two of the beloved characters, Michelangelo and Donatello, in real-time.

“We were looking for an innovative way to help socialize Nickelodeon’s Rise of the Teenage Mutant Ninja Turtles at San Diego Comic-Con this year, and the idea of doing a VR press junket seemed like a unique approach that would allow journalists and a select group of super fans to be fully immersed inside the reimagined universe with the ability to interact with the Turtles in real-time,” says Chris Young, SVP, Entertainment Lab, Nickelodeon.

Image courtesy of Nickelodeon.

Making live-action puppeteering possible

Leading up to the event, the team created puppets of the two characters in Adobe Character Animator CC that were meant to mimic what the characters look like and how they move in the new show.

“We worked with the executive producers of the show over the course of four or five months to get the animation right,” says David Gerhard, Creative Director of Alternative Media at Nickelodeon. “That was mission critical for us—we needed to align the interactive experience with the show to really give participants an immersive experience.”

The team appreciated the improved lip sync feature in Character Animator. The ability to use pause frames in folders also benefitted the puppeteers because it let them do more nuanced, expressive acting with the characters.

Once the puppets and animation were ready, it was time for action. Upon walking into a greenscreen room and putting on a VR headset, each participant was transported to an animated rooftop in Manhattan with a VR head of a popular Nickelodeon character on their body. The two animated turtles, Mikey and Donny, were there waiting for them.

David and his colleague Isaiah Kim acted as puppeteers, while the actual actors from the show, Josh Brener and Brandon Mychal Smith, provided the voices. Thanks to the intuitive and highly simplified controls within Adobe Character Animator CC, David and his team were able to focus on interactions with the audience, triggering character poses in real time using MIDI keyboards, without being bogged down by the technologies that made it possible.

Isaiah Kim and Dave Gerhard use Adobe Character Animator to puppet Michelangelo and Donatello of Nickelodeon’s “Rise of the Teenage Mutant Ninja Turtles Live” virtual reality experience. (Left to Right: Arielle Chongco, Isaiah Kim, Dave Gerhard, Youna Kang). Image courtesy of Nickelodeon.

“Having not done a ton of puppeteering myself, it was really interesting to just get in a zone of connecting with the voice actor,” says David. “You are anticipating their acting, and they’re playing along with what you might do. And that really made it fun for the interviewer who walked into the scene. It was super fun, and really exhilarating.”

Dave Gerhard uses Adobe Character Animator to puppet Donatello for Nickelodeon’s “Rise of the Teenage Mutant Ninja Turtles Live” virtual reality experience. Image courtesy of Nickelodeon.

The team used NewTek’s NDI technology to stream Adobe Character Animator into Unreal Engine, where the virtual environment took place. Foreground and background layers from the environment picked up over NDI were composited with the live-action footage in real-time in Resolume. At the end of the experience, participants walked away with a dynamic recording of their interviews.

“The idea for pulling this off originated from a meeting that I had with Adobe, Epic, and NewTek at NAB in Vegas,” says Chris. “I had built a demo using NDI to stream Character Animator into UE4 and was excited about finding a way to use the pipeline we had set-up to perform a live cartoon.”

Youna Kang, Caitlin Gillman, Marcus Velazquez, and Shazzy Gustafson use Resolume to composite the turtle puppets into their world while Isaiah Kim and Dave Gerhard use Adobe Character Animator to puppet the turtles. Image courtesy of Nickelodeon.

Building hype for the new show

Nickelodeon found ways to make the animation and storylines suitable for today’s audiences while staying true to the long history of the characters. The activation at Comic-Con played a key role in building hype for the newest iteration of the lovable superheroes.

“I didn’t really expect just how much fun people seemed to have once they were in the environment and talking to the turtles,” says David. “You sort of forget sometimes that an immersive VR experience with animated characters who can interact in real-time is still extremely new and can really get the imagination churning for anyone who participates.”

The Nickelodeon Entertainment Lab is working to identify other opportunities to use Adobe Character Animator CC for the show and elsewhere. With real-time animation growing as a trend in the cartoon industry, these types of activities and experiences are likely to become more common.

“There was a lot of creative and technical details that had to be sorted out to put this together, so it was super gratifying knowing that we had attempted something unique and were able to pull it off,” says Chris. “It really was a lot of different disciplines coming together. It was a hybrid of animation, game- development, live-action performers, puppeteers, virtual cinema photography techniques, on location in a live-broadcast setting. It wasn’t like anything we have ever done before.”

Be sure to check out Rise of the Teenage Mutant Ninja Turtles, now premiering on Nickelodeon.