Peek Behind the Sneaks: True Colors Shine Through with #ProjectScribbler

Cynthia Lu presenting #ProjectScribbler on stage at Adobe MAX 2017.

A good hand-drawn sketch artist can capture the reality of a moment with finely detailed work, the essence of a personality with a characature, or even communicate complex ideas with a simple storyboard. But adding color to these drawings is an often neglected step. When done improperly it can result in distracting, or even cartoony, looks. In fact, it takes special skill and a lot of extra time to add texture and color to a drawing in a realistic way.

Cynthia Lu, a research scientist at Adobe, wanted a better way to make color and texture accessible to every artist. The result is #Project Scribbler, a sneak technology at Adobe that uses Adobe Sensei to intelligently apply color and texture to hand drawn portraits.

“Our idea was to see if we could train deep neural networks to intelligently apply color gradients and texture details to hand-drawn sketches of people and objects,” Cynthia says.

That’s exactly what #ProjectScribbler does. On stage at Adobe MAX 2017, Cynthia demonstrated how Adobe Sensei can be applied to colorize simple line drawings, detailed sketches, and black and white photographs — simply by pressing a button.

#ProjectScribbler began as a collaborative project between researchers at Adobe and computer scientists at Georgia Tech and UC Berkeley. The team has published two different academic papers detailing the color and texture applications of the technology.

“Deep learning requires a large amount data to be able to make the best predictions, so we needed to collect a lot of training sketches. The challenge was where to find them,” Cynthia explains. “What we ended up doing was leveraging our own Adobe tools. We took tens of thousands of color photo portraits and used Photoshop filters to generate eight different styles of sketches from each one. Simulating a variety of hand-drawn styles is key for the network to correctly interpret real hand-drawn artwork of arbitrary style and fill in plausible colors.”

Correctly applying skin tone and hair color is a challenge the #ProjectScribbler team is working to address.

Another challenge the team faced was representing human diversity. “Even though we paid special attention to including a diversity of photographs — skin tones, hair color, gender, and even objects like glasses were all represented in the training data — the technology isn’t always capable of guessing the correct colors from a sketch,” Cynthia says. “Ultimately we’ll want to provide users with more specific control to indicate the desired skin tone, hair color, or even the right shade of makeup.”

Solving these kinds of technology challenges and putting more control into the hands of artists is what motivates Cynthia and keeps her passionate about her work. “We’ve reached the point where deep learning can be leveraged to not only recognize images, but to generate new and unique work. I’m really excited to see how we can leverage AI to give more freedom to artists and focus more on their creative concepts instead of technique,” she says.

Want to dig a little deeper into the technology powering Project Scribbler? Check out the story from Adobe Research.

This story is part of a series that will give you a closer look at the people and technology that were showcased as part of Adobe Sneaks. Read other Peek Behind the Sneaks stories here.