A (Software) Bug’s Life: How a Glitch in the Code Revolutionized Live Animation

Tooning out the News

All images courtesy of CBS Interactive.

In 2015, Adobe introduced Character Animator as a “Preview Release” or beta. The software uses facial tracking and lip-sync technology to turn webcam footage into an animated character using puppets, which are actually layered Photoshop or Illustrator files. These files can, of course, depict any type of character: an alien, a doctor, a monster, a president, or a cartoon version of yourself. Character Animator then transposes the movement and sound of the webcam capture and syncs it to the cartoon character, bringing real-time human expressiveness into an animated character — frowns, smiles, pauses, expressions of surprise, regret, and sorrow — the full spectrum, nuance, and spontaneity of human communication.

The Late Night Cartoons animation team at CBS’ The Late Show with Stephen Colbert was intrigued. Could a cartoon character be interviewed by a talk-show host, such as Stephen Colbert? They were sure audiences would be wowed by a cartoon character interacting naturally with a human host in a live interview setting.

Trying it out, they quickly discovered a bug in the lip-syncing capabilities of the software. Head animator Tim Luecke posted the feedback to Character Animator’s online product forum on Adobe.com. Within minutes, a Character Animator team member posted a response, kicking off a collaboration between Adobe engineers and the Late Night Cartoons team that is pushing the boundaries of “live” animation.

Fast-forward five years, and the team at Late Night Cartoons has taken Character Animator mainstream via the acclaimed “Our Cartoon President” character, which started out as a recurring segment on “The Late Show with Stephen Colbert” and soon morphed into the popular Showtime series, which is currently in its third season. And the technology itself won a Technology and Engineering Emmy Award in 2019, as a pioneering system for live performance-based animation using facial recognition.

Lead Tooning Out The News cartoon anchor, James Smartwood, comes to life in Adobe Character Animator (through a very human performance).

Launching “Tooning Out the News”

Now, the creative team behind “Our Cartoon President,” including executive producers RJ Fried and Tim Luecke along with “The Late Show with Stephen Colbert” executive producers Stephen Colbert and Chris Licht, has launched the new animated satirical news series called “Tooning Out the News” for CBS All Access, CBS’ digital subscription video on-demand and live streaming service. It features a cast of animated characters, lampooning top news stories as they break, and interviewing real-world guests. So far, these have included Alan Dershowitz, Mark Cuban, and politicians including Julián Castro, Rep. Ted Lieu (D-CA, 33rdDistrict), and Rep. Katie Porter (D-CA, 45th District). New five-to-seven-minute segments are available Tuesday through Friday, culminating in a weekly episode that showcases highlights. This means the animation team has to turn around broadcast-quality shows every 24 hours.

A few years back, the animation workflow implications for this type of show would have made it impossible. The “Tooning Out the News” team was confident that the technology was now in place to be able to produce high-impact animated segments, in a timeframe that could keep pace with today’s hyper-speed news cycle.

For executive producer Tim Luecke, it wasn’t difficult to sell the show to CBS All Access following the success of “Our Cartoon President.” “People were really excited for it when we suggested we could even do this. CBS All Access saw it could be a great sister program to ‘The Late Show with Stephen Colbert.’” The streaming service then showed its commitment to the cause by commissioning a bespoke studio for the show in the famed Ed Sullivan Theater in New York City, where “The Late Show with Stephen Colbert” is also recorded.

Tim Luecke created the show with Character Animator at the center of production.

Three days to lift off, COVID-19 hits home

After a setup time of four months, the team was confident that they had it all figured out — turning captured footage into beautifully rendered segments that would show a natural back and forth between cartoon interviewer and real-life interviewee. Then, the Friday before “Tooning Out the News” was to premiere, COVID-19 brought production to a halt. The entire team was sent home to work remotely and the premiere date was delayed.

For Technical Director Brian Maffitt, “When COVID-19 hit, we had to figure out a way to virtualize that whole thing and make it into a virtual version of a Live News Production Studio.” And the team had to prove to CBS All Access that they could still produce a show with the quality and consistency that today’s audiences demand. Working from their home offices, bedrooms, and basements, they produced four test segments in two weeks, and the streaming service gave the green light to go ahead and deliver the show.

The team collaborates on the rapid animation process from home studios.

With Animation Director Stephen Brooks, the biggest challenge was getting 20 or so remote-working animators to do character lip-sync and motion capture in the same way. The only way to make the schedule every day was to have multiple animators work on the same character to deliver consistency in how a character makes their mouth move to an individual sound or expresses an emotion. An “Oh” sound could be interpreted one way by one animator and another way by another. Described by teammates as “The World’s Foremost Character Animator Lip-Sync Artist™” with a knack, special talent, and “somewhat odd passion” for making lip-sync work, Stephen’s many Zoom calls and Slack conversations with animators helped drive consistency. Character Animator’s lip-sync capabilities are being powered by Adobe Sensei, and this machine-learning and artificial intelligence platform is now correctly anticipating many of the mouth movements that correspond to specific sounds.

And then there are the special requests.

Brooks explains: “In the morning we hear of any special things that might happen, such as a hand clap from a character — those usually take about five hours to finish.” The team tries to get those newly designed and hand-tooled requests done first so that when the video and audio comes in from the live interviews, they can move quickly to parse the lip-syncing and motion-capture syncing across the remote team of animators. If everything goes well, and it usually does, this work only takes three hours or so.

The digital puppets that the animators are working with and seen on screen are layered Photoshop files, with assigned layers for each behavior. An eyebrow movement is a layer and various mouth shapes make up a collection of layers for example, all of which bring unrivaled realism to the animated character’s vocalizing. The show uses about 20 puppets and some are cartoon caricatures of “The Talent,” including Jeremy Levick and Rajat Suresh, who play investigative reporters on the show and whose online videos have gained a cult following over the last year.

The one truism of working with technology in film and broadcast is that file sizes are enormous. Animation and video files are pixel hogs — and given the ubiquity of high-quality screens these days, they have to be. Obviously, with everyone working remotely, this presented a special challenge. Here another Adobe technology, Dynamic Link, came into its own. In the past, sharing media assets among post-production applications required you to render and export work from one application before importing it into another. Given the size of the files, rendering the same file multiple times, to get it just right, has the potential to be an enormous time sink. Dynamic Link offers an alternative to this workflow. The team uploads their Character Animator sequences and creates dynamic links into After Effects and Premiere Pro to show how the asset would look once rendered. This gives the team a huge amount of confidence that the files they send for rendering will not need substantial reworking — crucial for a daily show, with a quick turnaround.

Just a few weeks in, “Tooning Out the News” is making a splash with audiences and within the industry. And the team are quick to point to their close collaboration with Adobe as crucial to their industry-defining work. From executive producer Luecke: “None of this would have been possible if the Adobe team hadn’t responded to a frantic message-board post. The partnership developed over the last five years has been creatively wonderful. We’re in a unique situation, where we are trying to create a show that is at the nexus of animation, engineering, comedy, and news and we’ve got some of the top talent in all of those fields coming together to solve problems”.

“Tooning Out the News” is executive produced by Stephen Colbert and Chris Licht of CBS’ “The Late Show with Stephen Colbert,” and RJ Fried and Tim Luecke, who first collaborated on Showtime’s “Our Cartoon President.” The show is available exclusively on CBS All Access website and mobile apps.

At Adobe, we believe that everyone deserves respect and equal treatment, and we also stand with the Black community against hate, intolerance and racism. We will continue to support, elevate, and amplify diverse voices through our community of employees, creatives, customers and partners. We believe Adobe has a responsibility to drive change and ensure that every individual feels a sense of belonging and inclusion. We must stand up and speak out against racial inequality and injustice. Read more about the actions we’re taking to make lasting change inside and outside of our company.

We also know many people are still impacted by the current COVID-19 crisis and our thoughts are with you. The entire Adobe team wants to thank you, our customers, and all creators around the world for the work you do to keep us inspired during this difficult time.