Designing for Our Senses, Not Our Devices

I think it’s about time we take our relationship to the next level—our relationship with computing, that is. The way humans interact with computers and incorporate them into our daily life is shifting dramatically.

I explained in my last article that we’re at the event horizon of a new era of spatial computing—a world where digital experiences mesh with physical reality. Immersive, 3D technologies like AR and VR, along with voice and embedded sensors, are all converging into a new medium, powered by artificial intelligence.

It’s changing the nature of human-computer interaction. For the first time, we can use our voice, sight, gestures, and other natural inputs to directly connect with information. That, in turn, changes the nature of our relationship with computers. If we don’t have to think and behave like computers for them to understand us, and if computers begin to interact with us in more natural, human-like ways, the chances are we’ll stop thinking of them as machines and start treating them like our creative allies.

What is sensory design?

From a designer’s standpoint, creating the kind of future we want brings a new set of complications. Different inputs, surfaces and contextual needs all come into play. But more importantly, the designs we create result in more natural experiences that increase emotional connection—that’s a profound responsibility.

We need a new design language to move design beyond the confines of the screen and create natural, emotionally-engaging experiences that flow freely into the environment. I call it sensory design.

Sensory design is an adapted, humanity-inspired, industry-wide, design language for spatial computing. It’s the glue that connects digital interaction design to traditionally spatial disciplines like architecture, interior design and 3D modeling, as well as cognitive and behavioral sciences. It applies the skill and knowledge of UI designers, artists, scientists, activists, policymakers, engineers and system architects.

If that seems like an insurmountable skillset to acquire, don’t worry. Although I’ve found that designers, by nature, are curious, adaptive and really good learners, no single designer needs to do it all. Instead, we need to think about how we build teams and companies around the needs of sensory design. It requires diversity of skills, but also diverse teams with different backgrounds, cultures, ages and experiences—all need to be in the product design workflow to create truly human experiences.

Ana Arriola, general manager and partner for AI + Research & Bing at Microsoft, has over two decades of experience in product design, product management, and creative direction across new technology surfaces for consumer electronics, lifestyle experiences, UX/UI systems, and ethics and AI. She argues that sensory designers are part therapist, alchemist, chef and explorer. “We need to build teams with trust. Trust between product designers and others developing technology,” she says. “My perspective is that it’s best to have a deeply interdisciplinary team—creatives, data scientists, engineers, product designers, social scientists and legal—all working together to create experiences that are human, simple and authentic. All in service to humanity.”

What does it mean to design for the senses?

So, what does it look like to design for the senses, rather than the screen? It starts with a practical understanding of physical space and the realization that humans have excellent spatial memory. Most of us, for example, could easily close our eyes and walk about our house without an accident. We use our sense of proprioception to understand and encode the space around us, and designers should expect 3D designs to become the new normal.

But sensory design is more than spatial – it involves the rest of our senses as well. So, designers also need to plan for multi-sensory interaction, observation and analysis.

Understanding that many experiences will be placed into the physical world, designers need to consider how designs interact with changes in the physical environment—time of day, weather, lighting, temperature, traffic and other physical hazards and objects.

This also means we need to give up our sense of control. Design elements placed in the world cannot be controlled in the same way pixels on a screen can be manipulated. They have to adapt to environmental conditions, the physical dimensions, and the context of the surrounding environments. This means designers can no longer control the camera they view. Users are free to prescribe their own viewpoint, location, and context.

Perhaps most importantly, designers need to consider the emotional and psychological impact that emerges from genuine human interaction and collaboration. We can only do this by understanding fundamental human behavior, our bodies, and our cognitive abilities.

Ana has a wonderful conceptual model for considering these dynamics, based on the new Queer expanded variation of Chapman’s Five Languages of Love. It turns out that many of the ways humans express love serve as a useful guide for considering experiences in product design and spatial computing. Here are a few examples:

If spatial computing represents a new human relationship with technology, I find it profoundly hopeful to approach design from a framework of love and human understanding. That’s a clear path for taking our relationship with technology to the next level.