How AR, VR, and Voice are Redefining Digital Experiences

Emerging technologies like virtual reality, augmented reality, mixed reality, and voice interaction are reshaping the way we engage with the world and transforming digital experiences.

Imagine going to your favorite restaurant and watching social posts from your friends literally jump off the menu with recommendations for what to order. Or getting a history lesson in urban growth as you ascend a building in which the elevator’s walls come to life with immersive video. Or putting on a headset to experience life from someone else’s perspective — literally.

Digital experiences like these aren’t taking place in some far off, imaginary future. Many are happening right now — thanks to emerging technologies, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and voice interaction — which are reshaping the way we engage with the world.

Gartner predicts that by 2020, 30 percent of web browsing will be done without a screen, and by the end of this year, 10 million homes will have a room-based voice device like Amazon Echo or Google Home.

The trend is headed in a clear direction — digital experiences are moving beyond the screen. In the very near future, we’ll see business, creativity, work, and leisure move toward a world where your phone, laptop, desktop, and TV screens won’t be the sole conduits for digital interactions. New technologies are emerging to create deeper — yet less attached — connections between our physical and digital worlds.

“I think every surface has the potential to be a digital surface,” says Mark Asher, director of corporate strategy at Adobe. “Kiosks, immersive rooms, interactive walls and desktops, even our smartphones can be flipped into an immersive window that creates a 180- or 360-degree experience. All of this is becoming available around the world, and the world is being transformed into our own personalized digital canvas.”

The reality of moving beyond the screen.

While today we may be screen-obsessed with smartphones, laptops, tablets, and other devices, we’ll soon see more experiences that aren’t confined to a screen. AR, VR, and voice platforms are boundless — and the technology gives the space you enter the potential to become a more intuitive, efficient, and empathic canvas.

“I think the opportunity is to actually deliver the digital experience where it’s needed,” says Stefano Corazza, senior director of engineering at Adobe. “Instead of having to pull out my screen, and go to a URL to find the information about the store I’m walking past, I can just wear my glasses, and the information will show up right in front of me.”

Yet, immersive technology isn’t new. “The military, for example, has been projecting displays on pilot helmets for decades,” says Mark. “However, for other industries, VR and AR now have matured enough to provide use cases that make them reasonable for immersive experiences to be built into enterprise workflows. They can help workers get more work done, faster and safer than ever.”

Augmented Reality. Retailers, with their consumer focus, are often early adopters of new technology for business application. Nike customers can now design their own sneakers by slipping on a blank shoe. Then, using AR, object tracking, and projection systems, people can select custom graphics, patterns, colors, and sizes. Once the final design is saved, the shoe is produced onsite in 90 minutes. Nike is using AR to create a fast, fun, and interactive experience that stands out for its customers.

Nike customers can design their own sneakers with AR, object tracking, and projection systems. Source: Nike.

Virtual Reality. Unlike AR, VR creates an entirely new world that can be used to captivate users. Put on a headset and you’re transported to a place that is completely different from the one you physically occupy. Kayla Briët, a 20-year old Southern California filmmaker, composer, and VR artist, developed a very personal museum exhibit. It gives visitors time to reflect on memories that matter most in their lives by connecting them with VR access to treasured objects.

Voice technology. Voice-based platforms that are driven by artificial intelligence (AI) move people away from searching for content with their fingers, to using something much more intuitive and natural.

“For voice experiences, you need to have a natural language interface that performs well enough to understand different accents, dialects, and languages,” says Mark. “Now that the internet’s has been around for more than 20 years, we have enough data to be able to train voice assistants to provide this kind of narrow AI that allows people to actually have really good experiences with it.”

Additionally, advances in graphics processing, and software integrations, along with the shrinking size and cost of hardware, are bringing immersive experiences to life today.

“The ubiquity of personal and IoT devices containing low-cost sensors, such as microphones, depth-cameras, and GPS, alongside improvements in AI, is unlocking new possibilities for haptic touch, voice interaction, computer vision, and gesture recognition, as well as contextual, situational, and emotional awareness,” says Abhay Parasnis, chief technology officer and executive vice president at Adobe. “The human relationship with computing will become more natural, intuitive, and ‘invisible,’ which will, in turn, fuel new capabilities and usage models — everything from augmented reality to self-driving cars — further integrating computing into every aspect of our lives.”

Mark agrees. “We finally have the computing and the storage power necessary to deliver these experiences at a level of performance that is acceptable to us as human beings,” he says. “We’re able to suspend the disbelief that these experiences are artificial because we can now technologically deliver frame rates and the number of polygons necessary to actually make these virtual worlds look like truly immersive worlds.”

Taking off the rose-colored headset.

Innovations in technologies are driving real-time immersive experiences, but there’s still some progress to be made before we achieve the level of integration Abhay forecasts. There are important challenges companies must overcome on the road to widespread consumer and enterprise adoption.

The reality is, this is an evolutionary process, and the hardware we have today still has its thorns.

“The main issue I have with immersive experiences is that I have to pull out my phone and look around, and the headsets are still clunky at the moment,” says Stefano.

Aside from physically burdensome hardware, software options are fragmented and many require users to have experience with coding. While these developments offer a lot of potential from a creative standpoint, the tools remain difficult for creatives to work with.

As hardware improves, though, and immersive experiences become more commonplace, Mark is optimistic that our interactions with technology will improve. “I feel like we are on the cusp of finally bringing the humanity back to computing,” he says. “That means I’m now going to be freed to interact with digital experiences with my hands, or voice, or a nod — using my body and space like I do every day with other human beings — instead of this sort of artificial concept of having to click a button and roll a mouse around a desktop to perform every task.”

One issue to resolve before we reach that point centers around the difficulty of mapping images onto real-world objects, and getting an organic experience when space and time don’t hold still. AR relies on geographical coordinates to unlock additional layers of information in any physical space — essentially creating a virtual screen anywhere. Defining open standards for this type of mapping will help the possibilities move forward faster.

Another challenge is the difficulty of editing a 3-D video with 2-D tools. Adobe “sneaked” Clover VR at MAX last year to show how VR editing is evolving and becoming simpler to do — and now it’s headed into the latest release of Premiere Pro. It’s an editing interface for VR videos, in which editors actually use a VR headset to edit in real time. And voice technology is, itself, one way to make creative tools accessible for every audience. Lightroom Mobile, for example, is now outfitted with a digital assistant that lets people use voice commands to edit photos right on location — clunky hardware optional.

For other aspects of producing experiences beyond the screen, technologies have been hidden as niche resources. The right sound techniques, for instance, drive better experiences from a consumer perspective. Forty years ago, creatives began experimenting with Ambisonic audio to use sound in spatial contexts. Now, with the emergence of AR and VR, this niche resource is suddenly very relevant.

“​​We’re seeing physical design techniques — taken from film, theater, and industrial design — being applied to everyday use cases and accessed through our new spatial computing platforms,” says Silka Miesnieks, head of Adobe Design Lab.

Many companies are emerging to address the complexities of the AR and VR market, freeing it for exponential growth — Digi-Capital estimates that it will grow from $14 billion today to $120 billion by 2020. The opportunity that emerging artificial technologies presents is too great to ignore. Additionally, Gartner estimates that by 2019, 20 percent of enterprise businesses will evaluate and adopt AR, VR, and mixed reality immersive solutions, so it’s critical for companies to establish a game plan now, to avoid being left behind.

Get your hands on a new reality.

New beyond-the-screen technologies are evolving to democratize creativity and affect how we work. Adobe’s goal is to bring immersive technologies to market for a wide audience — whether designer, marketer, engineer, artist, programmer, writer, or anyone in between — and to remove the limits of what defines a canvas.

“We want a much broader audience to be able to create more AR and VR experiences than they are able to create today,” Stefano says. “If you’re looking at the immersive reality experiences available right now, 90 percent of them are created using a game engine, so that means people need to create using a desktop device, and they need to code. These two limitations alone will basically cut out most of the people on this planet.”

Stefano continues, “Instead, we want to allow people who are not necessarily coders, and not necessarily technical, to tell their story, and we want to allow those stories to be created on any digital device — particularly something as simple as a mobile device — and shared on any canvas.”

Adobe is researching artistic tools for VR with Project Dali.

Moving beyond the screen also indicates the need to identify new screens. While we imagine a world with limitless options, we are exploring new creative mediums. Adobe’s Project Dali, for example, is using VR to develop an endless creative canvas in the space around us. With custom brushes, artists see a three-dimensional image of their work as they paint, and they can examine it in real-time. The technology connects the digital tools to the physical world.

Sewn technology is a completely different take on a beyond-the-screen canvas. Instead of a visual canvas, technology is incorporated — often artistically — into fabrics and other media that combine to deliver a haptic experience — no screen needed.

Tell the story of the future.

With almost 1 billion phones being AR-enabled in the next year and VR headsets evolving rapidly, immersive media is poised to become the next disruptive platform. Widespread adoption of immersive technologies by both businesses and consumers will become standard tools sooner than we think. Silka believes easy-to-use, immersive design tools will empower innovation, and help the industry leap forward.

​​“One of the biggest mental leaps designers need to make is thinking about the user experience that lives outside the phone and in our 3-D world. Innovation will come from understanding human behavior and needs, and then applying immersive computing to address those needs,” Silka says.

Truly, technology alone won’t lead to immersive customer experiences. But telling an engaging story, enhanced and made real by technology, will. “It’s always about telling a story,” Stefano says. “Mixed reality is just another medium where you can tell a story that is more immersive, more personal, and more contextual.”

Companies and individuals that learn to marry compelling content with immersive technology will distinguish themselves from competitors, and create a sustainable business that helps redefine experiences amid the transformation of the Experience Era.

Read more about the future of immersive experiences in our Beyond the Screen collection.