Visualizing the Invisible

Adobe Design helps San Diego’s MOPA create a novel experience for museum-goers.

Image credit: Stacy Keck, MOPA.

Sparrows chirping around a bird feeder, the quiet hum of your refrigerator, your child’s giggle — the sounds that envelop our lives may be different, but for most of us, audio of some sort is ever present. And while sound waves constantly vibrate through the air around us, their physical form goes undetected to our eyes.

But what would it be like to see those sound waves? Adobe Design and the Museum of Photographic Arts (MOPA) in San Diego, California, took on this challenge together. The result is a unique hands-on exhibition at MOPA called “Note to Self.”

A photo booth with a twist

In this interactive installation, visitors create a self-portrait with the sound waves around them at that moment. They can manipulate audio and visuals in concert to create the portrait, which contains a layer of information that would be invisible without the aid of a camera — and they see the results in real time.

Taking cues from the phenomena of cymatics — the visible patterns that sound waves make — “Note to Self” uses liquid as an interface to visualize those sound waves. Visitors generate the portrait using a microphone and tone generator.

“The museum was planning an installation around the idea of what the camera allows us to see that we can’t normally see with the naked eye,” said Gabriel Campbell, designer. “We brainstormed a lot of different ideas and eventually became intrigued by cymatics. We realized that using sound as an interactive element could be a fun and engaging way for museum visitors to interact with a camera.”

Cymatics involves transferring sound waves through another material, such as oil or sand, and making interesting patterns in that material that change as the sound changes.

“We paired that idea with a version of a selfie station — playing off of the successful “Self-Composed” exhibit we created at the San Francisco Museum of Modern Art (SFMOMA),” said Ainsley Wagoner, creative lead. “We liked the idea of a photo-booth concept — both as a way to get several snapshots of the sounds being made as well as to create a fun artifact for visitors to take home with them.”

Image credit: Stacy Keck, MOPA.

Visitors can also access a digital version of their work.

The team decided to use a MIDI (musical instrument digital interface) keyboard as the source for the sounds. This allows any visitors who might feel shy about making noise to push keys to make sounds.

Adobe expertise

The project — a pro-bono effort for Adobe Design — required a team with a wide range of skills, including coding/bespoke software, engineering, graphic design, user interface and visual design, motion graphics, fabrication, installation, branding, and program/project management.

“The great thing about collaborating with Adobe was the team’s creativity and expertise,” said Joaquin Ortiz, director of innovation at MOPA. “Although we had worked on a few interactive technology exhibits like this before, Adobe’s team could see the project in a wider scope, with insight into the many ways it would connect with the public.”

The complex project took a year from initial meetings with MOPA to the exhibition launch.

“There were a million things to figure out,” Gabriel said. “We had to do a lot of tests to see if it was even a viable idea, write a lot of code to find the best way to merge the sound waves with the camera, source the actual equipment we’d use, and design the screens.”

A big challenge for the Adobe designers was to get the timing of the flow just right.

“We needed to learn, how many seconds does the user need to get ready? Is the countdown timer clear? How do they know that there will be three photos taken? How do they choose to print their photo strip?” Gabriel said. “We had a lot of trial and error, but we generated some great prototypes, so we could see all the different sequences put together.”

“The technology itself is a mix of OSX app development, OpenGL graphics shaders, web development (WordPress and general front-end work), and server-side development to connect with Amazon Web Services and administer the experience,” said Ben Farrell, lead developer.

This installation features a dish of liquid on a speaker in order to generate the visuals that respond to the sounds the user inputs.

“Filling a dish with liquid and having that liquid stay there and not evaporate, condensate, or leak while still responding to the sounds was a major concern,” said Will Ruby, experience designer. “In the end, our colleague Sam Wick found a chemical — silicon oil — that met our requirements, with the added benefit of looking more exotic than water.”

“Note to Self has been an overwhelming success,” Ortiz said. “Children are especially engaged by the interactive experimentation with the keyboard and trying different poses.”

“Our two organizations have developed a great relationship too,” said Gabriel. “We hope the visitors enjoy the experience and come to think about science and photography in a new way.”

Special thanks to all the volunteers who helped with this exhibit, but didn’t appear in this story:

Ainsley Wagoner, senior experience designer
Amy Casillas, senior experience designer
Anny Chen, senior experience designer
Ben Farrell, senior experience developer
Danielle Morimoto, experience designer
Emma Zhang, experience designer
Gabe Campbell, senior experience designer
James Stallmeyer, senior experience designer
Jeff Booher, senior computer scientist
Lisa Pedee, program manager
Sam Wick – Sr Experience Designer
Samarth Gulati, web UI and prototype developer
Will Ruby, senior experience designer