AI Meets AR

Augmented reality is now even more intelligent by design.

Before/after of Match Image in Adobe Dimension.

You’re considering buying a couch online, but, despite having measured and remeasured the dimensions over and over, you’re concerned. What if it doesn’t look right? What if it completely takes over the room? There’s a barrier to the experience — distance, viability, or simply not being able to get the couch in your living room until you’ve committed and clicked “Buy now.”

These are the customer challenges facing today’s brands. People demand relevance and value — and that means creating the kind of meaningful, immersive experiences that help them definitively decide on that new sofa.

Enter augmented reality — and artificial intelligence.

As technology evolves, so does the way brands connect with consumers. Augmented reality (AR) is quickly becoming one of the biggest game-changers for businesses, profoundly transforming brand engagement.

To create these compelling and powerful customer-centric experiences, AR doesn’t act alone. The underlying technology that makes the new dimensions and immersive experiences possible is artificial intelligence (AI).

The power of AI + AR

AI is at the heart of AR platforms, making the two well-positioned to sync on a variety of applications.

“I believe AR is a game-changer,” says Stefano Corazza, senior director of engineering at Adobe. “I think it allows people to make the world incredibly playful. The digital is really lending a hand to the physical, and they are leveraging each other’s best aspects and best features to make the most meaningful experiential journey.”

AI is the key to enabling AR to interact with the physical environment in a multidimensional way. Object recognition and tracking, gestural input, eye tracking, and voice command recognition combine to let you manipulate 2D and 3D objects in virtual space with your hands, eyes, and words.

AI enables capabilities like real-world object tagging, enabling an AR system to predict the appropriate interface for a person in a given virtual environment. Through these and other possibilities, AI enhances AR to create a multidimensional and responsive virtual experience that can bring people new levels of insight and creativity.

The result is mixed reality in a single physical environment — seeing that sofa in your living room before you click “Buy.” Through examples like this, AI can interpret and enable AR platforms to act on real-time, real-world environments. The end result is more relevant, and delivers more immersive experiences that delight and enlighten the customer journey.

These journeys are becoming increasingly common. AR is quickly becoming part of mainstream culture and AI is already well into its “revolution,” powering countless examples of everyday technology. From identifying and filtering email spam, to finding similar Pinterest “pins,” to deciphering handwriting on checks being deposited via mobile, AI is highly integrated into our days, whether we recognize it or not.

This, though, is just the beginning for both AR and AI. The next step is for businesses, creatives, and experience makers to understand trends and best practices surrounding AR — and, from there, use the power of AI to produce innovative content and consumer experiences. In doing this, brands and content creators can deliver more relevant, immersive, and compelling experiences for a broad customer audience.

AR experiences are already bubbling to the surface in a number of industries and niches, including retail, art, and education. Though diverse in their approaches and applications, these industries are leading the AR/AI charge and, with it, creating incomparable engagement opportunities for their consumers, students, and users. Here’s how:

Retail: Redefining “try before you buy”

The latest AR technology has led to new paths for consumer interaction and pre-purchase experiences. Recently, furniture retailer IKEA tapped into ARKit to create IKEA Place, an iPhone/iPad app that lets users virtually place furniture in their homes. By taking this try-before-you-buy approach, potential buyers can see these large items in their intended spot before making a major purchase — a purchase that would be hard if not impossible to return.

Using the app, users can snap a pic of their room, then drag and drop 3D and true-to-scale models of IKEA furniture into the image. AR anchors the immersion — seeing your room with the new piece front and center — while AI capabilities support the multidimensional manipulation, creating a powerful experience. This provides an instant and accurate sense of how the piece would look and feel in the space.

Similarly, L’Oréal has created an AR app on Facebook. The brand recently acquired Modiface, which developed the virtual makeup app. Like IKEA, this app allows users to try before they buy — while cosmetics aren’t as big or costly as furniture, they often can’t be tested before purchase or returned after purchase. AI is used in the app to track the user’s face and hair in the image to enable realistic application of virtual makeup and hair dye.

In letting users test-drive products, L’Oréal gains a critical competitive advantage from an experience perspective. “From a brand experience perspective, the convergence of AI and AR is transformative for businesses,” says Stefano. “The brand experience becomes more whimsical and more fun.”

Art: Increasing the impact

The art world has begun exploring AR, reimagining both how artists create and how audiences experience and appreciate art, transforming viewers’ experiences.

At the San Francisco Museum of Modern Art (SFMOMA), a recent AR experience featured a gallery equipped with interactive artwork inspired by René Magritte. Once inside, visitors could insert themselves into a series of paintings, mixing their images into the art. The goal here was to prompt questions about reality and representation.

“These kinds of wordless, playful experiences can have a tremendous impact,” says Chad Coerver, SFMOMA’s chief content officer. “It somehow serves to cement the experiences and bring them closer to the artwork in ways that words can’t.”

Stefano agrees that one of AR’s strengths is its ability to engage an audience on a deeper level than audio-visual communications typically do, delivering a much stronger sense of empathy.

“I think there’s an inherent quality of the AR tools that create more emotionally impactful experiences,” he says. “It feels so real. One of the immersive media artists in our Creative Residency program defined AR in the best possible way. He said, ‘AR is a collective hallucination.’ So our goal at Adobe and as makers of tools is to make that hallucination feel as real as possible.”

Education: Surrounding students with every lesson

Education is another field just beginning to delve into both AI and AR — with unparalleled opportunities surrounding personalized and experiential learning. Many have declared this technology will “change education forever.”

AR has particularly broad applications for experiential learning. Using this technology, students can interact with virtual realities, providing new perspectives and an ability to gain more sophisticated appreciation through “hands-on” learning.

For example, Visible Body is an AR human anatomy atlas app that turns anatomy education into an interactive experience rivaled only by a cadaver lab. By visualizing and interacting with a 3D, life-size, holographic version of the human body, users become intimate with every aspect of anatomy in a way that books and lectures simply can’t match.

It’s educational experiences like these and countless others that are giving students an opportunity to explore, interact, and create unparalleled experiences within a simulated world, according to “Intelligence Unleashed,” a recent study on AI in education — and those experiences are opening the door to unparalleled real-world learning poised to redefine the classroom and beyond.

Democratizing content creation

In these and countless other examples, AI and AR work instep to create new and better content, and creative and immersive experiences — experiences that have become increasingly critical to designing engaging customer journeys.

Adobe is leading the way, with Adobe Sensei putting AI at the center of designers’ toolkits and Project Aero enabling exploration in immersive media. Technology like this allows enterprises to design richer, more intuitive, and more seamless AR experiences — experiences more relevant to their diverse audiences.

“Your imagination is the limit, but that creativity has always been living on a screen,” says Stefano. “Now AR and AI are working together to bridge these two worlds. Now, your imagination is the limit.” That said, adoption and democratization still need to happen to make these experiences par for the course, he adds.

Also central to this evolution is Adobe Dimension, with features powered by Adobe Sensei. With Dimension, people can create product mock-ups, brand visualizations, packaging designs, and render photorealistic images with intuitive, easy-to-use 3D tools.

Dimension’s Adobe Sensei-powered Match Image leverages AI to insert virtual objects into real-world scenes. While Project Aero does it with a live camera feed, Dimension uses a single photo, instantly aligning and lighting a 3D composite quickly and easily.

Match Image in progress.

“Thanks to Adobe Sensei, Match Image not only speeds up digital 3D creation that previously required expensive photo shoots or laborious model creation,” says Ross McKegney, director of engineering for Adobe Dimension, “but also democratizes creative compositing workflows that were previously only available to those with extensive 3D training.”

Also unique to Dimension is Magic Wand, which uses Adobe Sensei machine learning to enable people to select sections of a 3D model that are not already separated. This makes selection faster, smarter, and more efficient, making subsequent interactions, such as changing the material of part a 3D model easier. The Magic Wand tool works by analyzing shape, curves, edges, and symmetry.

There’s also the Sensei-powered Place Graphic feature, which simplifies the 3D design process even more. “Placing a graphic on a 3D model typically requires a number of steps to account for the nuance of 3D texturing and other qualities — things like UV mapping and creating 2D textures that then are rewrapped on a 3D object,” says Ross. “The Place Graphic feature in Dimension eliminates these steps, so the designer can work much more quickly and intuitively.”

“It’s our mission to democratize this kind of content creation,” Stefano says. “The biggest challenge that we have ahead of us is, how do we define an efficient and intuitive human-computer or human-machine interface? When will we all be wearing AR glasses? When will AR be that pervasive? And, again, how do we reinvent the human-machine interface so we can create those experiences in a very fluid and intuitive way?”

Read more about how AI helps deploy new technology at scale, and find out more about Adobe Sensei, Adobe’s AI and machine learning technology.