Removing window reflections in Adobe Camera Raw
A snapshot that is ruined by reflections from a glass window (left), the photo with the reflections removed (center), and the reflection itself (right) of a backyard garden.
Glass windows are one of mankind's most useful inventions, allowing us to live in a wide range of climates by insulating us from heat, cold, noise, or unwelcome visitors like insects, yet providing a clear view of whatever is beyond the window.
But windows are subject to reflections, which makes photographing through them challenging. Reflections occur when you're on the inside of vehicles, offices, or kitchen windows looking out, especially if the space you're in is brightly lit. Reflections also occur when you're on the outside looking in — through living room windows (like the example above), storefronts, display cases, or the protective glass over a famous painting.
Reflections are sometimes useful. For example, they help you distinguish between different materials. However, reflections that show recognizable objects are typically unwanted and get in the way of photography. Indeed, we often give up taking a unique photo that we expect will be ruined by a reflection. So while few of the photos in our collections might contain unwanted reflections, this is mainly because we passed up many creative opportunities. A common example is photographing aerial scenes through airplane windows, a perspective hard to achieve any other way, but often polluted by reflections of the cabin interior.
In this blog we describe Reflection Removal, a new technology that can eliminate reflections from photographs taken through windows with a single click. Our technology is powered by AI, but it's not generative AI. This first iteration of the tech is designed to address only one kind of reflection — from plate glass windows that cover most or all of your field of view. It's not designed to remove reflections from windows that are small or far away, or where the window frame is within the field of view, or reflections from objects like wine glasses, car bodies, or clouds reflected in a lake. We might address some of these applications in later updates. Our goal is to help you turn a photo you might otherwise delete into one that is good enough to share. Reflection Removal is available now in Camera Raw as a Technology Preview, to get feedback from the community, and will be coming soon to Lightroom.
How does it work?
Given a photograph that contains window reflections, it's not obvious how to remove the reflections — or if it is even possible. Let's start by taking a brief detour into the physics of reflection.
A photograph that is polluted by a reflection is the sum of two images. The first is the scene you'd like to photograph — on the other side of the window. The second is a reflected view of the scene behind you, weakened because glass reflects only a portion of the light that hits it. In the example below, the captured photo is the sum of an image of turntables photographed through the front window of a music shop in Zurich, and a reflection of the cobbled street and the photographer.
Interestingly, the second image is not the same as the view you would get by turning the camera around and pointing it at the street. After all, the reflection does include the photographer! Instead, it's the view you would get by placing the camera inside the store, then turning it around (and removing the glass). Technically, we say that the second image corresponds to a virtual viewpoint the same distance behind the window as your camera is in front of the window.
So that’s how reflections form, but what do they look like? Because windows separate different kinds of scenes (like indoor and outdoor), the images being summed by the window typically have different scene content, white balances, sharpness of focus, and so on. Also, in photos that contain reflections, if two edges cross, like the edges of the turntable and the legs of the photographer, it's probably due to a superimposed reflection, because the edges of opaque objects don't cross in nature.
The AI model behind Reflection Removal identifies and untangles these two images. We train the model by gathering thousands of photographs of varying subjects that contain no reflections. We then add pairs of them together to form millions of simulated photographs that appear to be polluted by reflections. An example is shown below:
One of our training examples. Ordinary photographs of an outdoor scene and an indoor scene are added together to form a simulated image (right) polluted by reflections. Our Reflection Removal model learns to separate the third image into the first and the second, with the two original images serving as ground truth.
We then run each simulated image through the model and ask it to predict what the two originals were. We know the real answer, so we can reward the model’s successes and penalize its mistakes. By repeating this training process over many examples, the model learns how to separate the two images in a photograph that is polluted by reflections.
How should you use this feature?
For now, our technology works only on raw photos (DNGs, CR2s, ARWs, ProRAWs, etc.) To try it on your own images, open the Camera Raw plug-in and go to the Technology Previews section of the Preferences Panel. Then enable the New AI Settings and Features Panel and restart the host application (Adobe Photoshop or Adobe Bridge).
Once you import your photo, go to the Remove panel (the eraser icon along the right side of the user interface), and in the Distraction Removal section, check the box labeled "Reflections." The model runs in a few seconds and gives you its best prediction of what the scene looks like without reflections.
When the model is finished, the slider below the checkbox will be set to 100. If you move it back to zero, your reflections will gradually reappear. Keep moving it to the left, to -100, and you'll see the model’s prediction of the scene that was reflected in the window. You may be surprised at what it reveals!
The animation below gives an example of moving the slider. The scene is the display case in Marc's Stanford University office, and the reflected scene is Marc taking the photo using a smartphone. Sometimes our model mixes up the scene behind the window and the reflected scene — that's why we give you a two-ended slider. But because our model is not generative AI, it will never create objects that weren’t present in the original photograph.
This video shows the effect of moving the slider after removing reflections from a snapshot of a display case containing objects from the history of computer graphics. Aficionados might recognize the terra cotta Stanford Bunny on the top shelf.
Some more examples
To help you understand when Reflection Removal is likely to work well, and when it might not, let's look at a few more examples. These are everyday snapshots, not professional photographs, and all were captured using smartphones. To see higher-resolution versions of these examples, and for more examples, see this album.
The Swiss village of Disentis (or Mustér in Romansh) photographed through a train window. In this case the reflection is completely removed, along with a haze caused by dirt on the window.
Mount Rainier and other volcanos of the Pacific Northwest photographed through an airplane window. In this case, removing the reflection also attenuates the effect of atmospheric scattering, which is a form of reflection.
Poster advertising a neighborhood movie theater in San Francisco, photographed through a protective plexiglass sheet. The recovered reflection shows the photographer and the street scene behind him. Our model also works well for removing reflections due to the glass that protects paintings in museums.
Here’s an unexpected use: photographing a tidepool along the California coast. It’s hard to take a picture like this without your camera (or smartphone) being reflected in the water surface.
A failure case. The reflection in this photograph looking into a garden store is so strong that even you, dear reader, will struggle to disentangle the two scenes.
Some pro tips
The best way to remove reflections from a photograph is to avoid them in the first place. Indeed, if a reflection is so strong or complex that a person looking at the photograph struggles to figure out what is what, then our model might struggle as well. For photographs looking out of rooms, try turning off the room lights, making it darker where the camera is than the scene you're photographing. If you're photographing into a display case, press the camera lens against the glass, using an ultra-wide lens if needed. For a museum painting lit by overhead spotlights, stand back from the painting and use the telephoto lens — this often suffices to move the reflected spotlights outside the frame of the painting. If this doesn't work, shoot from the side — you'll get a distorted view, but you can fix this in post using Camera Raw’s Geometry tools.
If you're forced to deal with a reflection, the physics of reflections says that shooting head-on into the glass produces a weaker reflection than shooting from an angle. Alternatively, you can shoot from an angle and use a polarizing filter, but this may change colors in an unwanted way. Finally, try to avoid reflections that are so bright that they saturate your camera sensor, like the sun or artificial light sources. If pixels are clipped to white, the model has no idea what those pixels might look like without the reflection. In these areas, we'll insert something dark or noisy, but we won't try to guess what might have been there. You can use Photoshop’s Generative Fill to replace these areas with generated content if you wish.
Let's talk briefly about workflow. It doesn't matter whether you remove reflections before or after you crop or adjust your photograph because our model operates on the original uncropped image without adjustments. If the photograph is oriented incorrectly, e.g. rotated 90 degrees, you should fix this before removing reflections, or you might confuse the model. After removing a reflection, you might want to tweak the white balance, Blacks, Dehaze, or other controls, since you're now looking at a different photograph! And if you invoked the Adobe Adaptive Profile you should update it — otherwise, the profile might leave traces of the reflection in your result. Don't worry, Camera Raw will remind you to do this.
Shiny spots and rough edges
Reflection Removal was built by Eric Kee and Adam Pikielny, with help from Jiawen Chen, Lars Jebe, Durga Ganesh Grandhi, Eric Chan, Thomas Knoll, Simon Chen, Frieder Ganz, and Kevin Matzen. We believe our technology is the best-performing feature of its kind, so we're excited to see what you do with it!
That said, removing window reflections is a hard problem, and this is our first stab at it, so there are inevitably some rough edges. For example, we don’t currently do well on cityscapes at night. In fact, removing reflections is what mathematicians call an ill-posed problem, meaning that for a given photograph it's not possible to decide with certainty which objects are in the original scene versus the reflected scene. One polluted photo could have many plausible separations. The key is training our model to understand scenes that are likely to exist in the real world. We're constantly improving the model, so stay tuned!
Sometimes the model will remove a reflection you like — if so just pull back on the slider a bit. A more complicated option is to import the DNG into Photoshop three times, with the slider at -100, 0, and 100, load them into layers, change the Image Mode to 32 Bits/Channel, which makes pixel values proportional to scene brightness, and create masks around regions you want to save in one or more of the images. If you're blending together the -100 and 100 images, use the "Linear Dodge" blending mode — if you're blending either of these against the original, use the "Normal" blending mode. Here’s an example:
What's next?
We're planning to support JPEGs, HEICs, and other non-raw files. We're also looking into removing small reflections like eyeglasses and distant windows. We'd also like to extend our tool to the removal of dust, scratches, rain, snow, or other things that land on windows (bugs on windshields?) Finally, while this beta is available only through Camera Raw, we plan to bring an expanded version to the entire Lightroom ecosystem.
As with all our products, we’re always looking for feedback from users. A good place for this is in our public forum. We'd love to see the results you get removing reflections from your photographs. We'd also like to hear your suggestions for follow-on features. Try it out and let us know what you think!