Moving Beyond Keywords to Make Visual Search Smarter: How AI Understands Creative Intent
Most visual creations start with a search—for images, colors, fonts, and inspiration—but search has always felt disconnected from the creative process. It can be tedious and time-consuming to translate brilliant, imaginative ideas into words. Search terms rarely convey the aesthetics and emotions at the heart of a creative idea – which can make image search become a mind-numbing task when it should be inspiring.
That’s why we’re using our artificial intelligence (AI) and machine learning technology, Adobe Sensei, to fundamentally change the nature of search and help make creative visions a reality.
What AI-powered search can do, and what it means for creativity
With deep learning, we’re training our search algorithms to better understand images to recognize objects—like cars, cats, humans, or even the Eiffel Tower—as well as colors, composition, style, and mood. Then, we break it all down so people can search for any of these aspects, or combine search terms and components from multiple images.
The result is a search that doesn’t just find something similar to a keyword or image—it finds something similar to the exact elements that are most meaningful to the searcher. Ultimately, search captures nuance and creative intent.
Search results in Adobe Stock showing images based on a “sunset” query that also includes a specific color palette.
Here’s an example of how it works in Adobe Stock. Imagine you’ve just done a keyword search for an image of a gazelle. You have a trove of interesting gazelle photos, but none of them is quite right. To hone in on what you need, you choose an image that’s close. Then, you can use image similarity search to find gazelle photos with similar colors, content, and composition to your original. You find the perfect lone gazelle, looking to the right, with a warm color palette.
But what if gazelles aren’t the only piece of your puzzle? You might also need a German shepherd in the same pose and location, so you change your keyword, but keep the composition constraints, which quickly takes you to images of a German shepherd that look remarkably like your gazelle—your dog is front and center, looking right with warm colors.
Image search in Adobe Stock for the term “gazelle”
“Find similar” image search results in Adobe Stock of a gazelle looking to the right, based on a sample image’s attributes (content, color and composition)
“Find similar” image search results in Adobe Stock for a ‘German shepherd’ query based on an image of a gazelle looking to the right
As you can see, the combination of keywords with visual search for subtle aspects of an image allow the search engine to understand the nuances of your aesthetic and creative vision—far beyond what it could ever do with keywords alone.
Now, imagine that you want an image with a particular kind of object. Maybe you have a photo of a tent in the woods, but you want the tent further to the left, and out of the woods. You can click the tent, an object that machine learning technology can recognize, slide it where you need it, and search images with a tent in exactly the spot and scenery you desire. Eventually, you will even be able to engage several images in your search, choosing an object from one, the colors from another, and the composition from a third.
Image search in Adobe Stock for the terms “tent” and “forest”
“Find similar” image search results in Adobe Stock of a tent in the forest, based on a sample image’s attributes (content, color and composition)
“Find similar” image search results in Adobe Stock of a tent in the forest, based on a sample image’s position and size of the tent – with the tent becoming the focal point of the image
“Find similar” image search results in Adobe Stock of a tent in the forest, based on a sample image’s position – with the tent being displayed on the left side of the image
Looking ahead: How search will create the things we imagine
The journey with AI and visual search is just beginning. As the tools like Adobe Stock become even smarter, search and creation will come together in a complimentary way. Eventually, AI won’t just help us find things, it will generate what we’re actually seeking. The evolution of smarter search is like a dance—humans are creating the data from which AI learns. Smarter tools, combined with the indelible spark of human creativity, will usher in new avenues for creative expression that we haven’t previously imagined.
For example, imagine you’re searching for an image of person with an umbrella on a sunny day, but all the images you find are rainy days. Search will be able to apply machine learning to blend assets and create an image that never existed before—an image that’s exactly what you had in your imagination. Think of layering the magic of Photoshop to your search query and applying that to find the image you envisioned.
For creatives, visual search harnessing the power of AI means cutting out a lot of the grunt work and the frustrating, clumsy keywords. By doing this, the search becomes more deeply integrated into the creative side of the work, and people have more time free to devote to the innately human side of creativity—developing and exploring new ideas.
Portions of this article originally appeared on PetaPixel.