Denoise demystified

“From the ACR team” is a blog series that brings you insights directly from the team that builds the imaging features for Lightroom, Lightroom Classic, Lightroom mobile, Adobe Camera Raw, and the Camera Raw filter in Photoshop. I recently worked on an AI-powered feature called Denoise and it’s available today. I collaborated closely on this project with Michaël Gharbi and Bo Sun, who developed the cutting-edge machine learning models behind this feature. Michaël also previously developed two related features, Raw Details and Super Resolution.

I created this photo early in the morning. I used a high ISO setting of 6400 because it was dark, and I was photographing hand-held from a moving boat. The zoomed-in crop (below left) shows that there is a lot of noise in the original capture, but the new AI-powered Denoise feature in Lightroom and Camera Raw has no trouble cleaning it up (below right).

Denoise is now shipping in Camera Raw 15.3, Lightroom 6.3, and Lightroom Classic 12.3. In this post I’ll explain what it is, how it works, and how to get the most from it.

All about noise

Noise is an integral part of photography. We’ve always had noise in some form or another, starting in the early days with film grain. In fact, we’ve been seeing noise and grain for so long that they’re part of our visual language. A bit of noise is not an issue and can actually make a photo look more natural. Too much noise, however, can overwhelm the photo and make it hard to see clearly.

There are many sources of noise in photography. A major source is the nature of light itself: recording light is a fundamentally statistical and noisy process, even if we had perfect cameras. To understand why this is, imagine it’s raining and you’re collecting rainwater with buckets. You place two identical buckets side by side, wait for an hour, then measure how much water each bucket holds. If you repeat this experiment lots of times, you’ll find that the two buckets have the same amount of water on average, but that for any given experiment, the exact amount of water in each bucket varies a bit. That variation is the noise. Similarly, the pixels on a sensor don’t all capture the same amount of light; during exposure, the number of photons landing on one pixel isn’t exactly the same as the number landing on its neighbor.

Sensors themselves also contribute some amount of noise from the readout and processing circuitry, mostly noticeable in the shadows. Sensor noise tends to get worse at higher temperatures, including long exposures.

Noise may be a fact of life, but that doesn’t mean we have to live with noisy photos. This is where Denoise comes in.

Introducing Denoise

It’s been ten years since we last updated our noise reduction algorithms in Lightroom and Camera Raw. Back then, most cameras were 8 to 12 megapixels, and the highest ISO setting available was 25600. Lots of changes have happened since then:

• Sensor technology has advanced greatly, with ISO 25600 now part of the “standard” range in many cameras.
• Phone cameras are now everywhere. They are incredibly convenient and offer unique capabilities, but their sensors are necessarily small and therefore more sensitive to noise.
• The research community has made significant advances in developing machine learning (ML) methods to reduce noise.
• Many computers now have dedicated hardware for running those ML models, making it practical to use those fancy new denoising methods on real photographs.

Denoise is our third Enhance feature in Lightroom and Camera Raw, following Raw Details and Super Resolution. It’s also by far our most ambitious and advanced effort. We had some very specific quality goals when developing this feature: natural-looking results with crisp edges, clean shadows, good preservation of texture and small colors, and minimal artifacts (like uneven splotches). As a rule of thumb, we wanted to deliver clean, usable results for a 20 megapixel full-frame camera at ISO 51200. That’s a high bar!

Here’s a furry example of the quality Denoise can deliver:

I photographed Elmo sitting in his favorite box. It was almost completely dark, with the only light coming from the room behind me, so I had to crank up the ISO to 30000 to get a usable shutter speed. Not surprisingly, the original photo shows a lot of noise (below left). Denoise does a good job taming the speckles while retaining the texture in Elmo’s fur and the details and colors in his eyes (below right).

How does it work?

Camera sensors see the world through mosaic patterns like the ones shown in the illustration below:

Our previous blog post on Raw Details explains more about these patterns and how we obtain high-quality RGB images from them.

Denoise uses machine learning to interpolate those patterns and remove noise at the same time. That is, our models are designed and trained to perform both demosaicing and denoising in a single step.

Michaël Gharbi and Bo Sun developed the core technology behind Denoise. As with our previous Enhance features, the idea is to train a computer using a large set of example photos. Specifically, we used millions of pairs of high-noise and low-noise image patches so that the computer can figure out how to get from one to the other. Here’s what some of them look like:

These are small crops from detailed regions of real photos. They contain material from everyday life. Bricks and branches, trees and twigs, cloth and fabrics, and so on. With enough examples covering all kinds of subject matter, the model eventually learns to denoise real photos in a natural yet detailed manner.

Teaching a computer to perform a task may sound complicated, but in some ways it’s similar to teaching a child — provide some structure and enough examples, and before long they’re doing it on their own. In the case of Denoise, the structure is called a “deep convolutional neural network,” a fancy way of saying that what happens to a pixel depends on the pixels immediately around it. In other words, to understand how to up sample a given pixel, the computer needs some context, which it gets by analyzing the surrounding pixels. It’s much like how, as humans, seeing how a word is used in a sentence helps us to understand the meaning of that word.

We developed a few extra goodies in our training system for Denoise. A key ingredient is that we developed an extensive noise simulation and data augmentation pipeline so that the resulting models would be robust and work in a broad range of real-world situations. Another essential component is the use of a large data set of “dark frames” which helps the model to understand and remove pattern noise in the shadows, especially in older cameras. (Essentially, my teammates and I spent a lot of time recording images with the lens cap attached; never before have so many gigabytes been spent on such underexposed images!) Third, as with our previous Enhance features, we continued to train directly from the raw data, which enables us to optimize the end-to-end quality. In other words, when you apply Denoise to a raw file, you’re also getting Raw Details as part of the deal. Finally, we built our machine learning models to take full advantage of the latest platform technologies, including NVIDIA’s TensorCores and the Apple Neural Engine. Using these technologies enables our models to run faster on modern hardware.

How do I use it?

Using Denoise is easy:

1. Click the new “Denoise” button in the Detail panel to bring up the Enhance dialog.
2. Adjust the Amount slider to taste.
3. Press the Enhance button.

After multiplying and adding up a gazillion numbers, your computer will produce a new raw file in the Digital Negative (DNG) format that contains your denoised photo. As with previous Enhance features, any adjustments you made to the source photo will automatically be carried over to the enhanced DNG. You can edit this DNG just like any other raw photo, applying your favorite presets and custom tweaks.

In the Enhance dialog, press-and-hold within the preview image to see the image without Denoise applied. This is a convenient way to make a before/after assessment, especially when fine-tuning the Amount setting.

Applying Denoise will automatically apply Raw Details, too. Combining these steps results in higher quality and faster performance.

What about the previous Noise Reduction sliders, like Luminance and Color? They’re still here, but they’re tucked away in a new sub-panel called Manual Noise Reduction. Note that Denoise will automatically set these sliders to zero on the new DNG.

Denoise is currently supported only for Bayer and X-Trans mosaic raw files, but we’re looking into ways to support other photo formats in the future.

Examples

Let’s go through some examples, starting with this studio scene from DPReview captured at ISO 51200.

It’s not a “real-world photograph” but don’t worry, we’ll get to those soon enough. A studio scene like this, with lots of small details, textures, and various colors, is a very good way to get a sense of Denoise in action. Here are some before/after crops of different sections. Keep in mind that the photo was captured at ISO 51200!

Now let’s look at some “real” photos. Wildlife and sports photography often involve high ISO settings to maintain a sufficiently high shutter speed for capturing a fleeting moment. That was certainly the case for the photo below, where I needed ISO 6400 to record this interaction between two polar bears in fading light:

My capture is also somewhat underexposed, so I brightened it afterwards with Exposure +1 in Camera Raw. Unfortunately, this makes the noise quite visible (below left):

I applied Denoise at Amount 50 (above right) to clean it up. This gets rid of the color speckles yet retains detail in the bears’ fur and color in the teeth. And my, what large teeth they have!

Photographing indoors (at home, in a restaurant, or in a concert hall) is another common case for using high ISO settings because of low light. I captured the following scene a few minutes before Daniil Trifonov walked out on stage and breezed through Liszt’s Transcendental Etudes (yes, all of them!):

Let’s take a closer look at some of the details behind the piano:

The original (left) shows a lot of colored noise, but a touch of Denoise (right) tames it while properly distinguishing the warm colors among the ornate structures.

Let’s see how Denoise stacks up against our previous (manual) noise reduction sliders. I photographed this sunburst while hiking in a forest in northern California.

I didn’t have a tripod with me, so I captured this freehand at ISO 12800. The original capture (below left) has a lot of noise, which is very apparent in the foliage and bark:

Manual noise reduction (above middle) eliminates the color noise, but it struggles to maintain color separation between the green foliage and the brown bark. It also loses some of the bark texture. Denoise at Amount 90 (above right) does a better job of keeping the browns and greens separate and preserving the bark texture. Here’s another side-by-side comparison of another section from the same photo:

Again, Denoise (right) does a much better job at retaining the texture of the bark compared to the manual sliders (middle).

Night photography is another case where high ISO settings are unavoidable:

The original capture shows plenty of noise in the sky (below left). Manual noise reduction does a good job removing the noise (middle), but it also removes a lot of the natural color variation among the stars. The new Denoise (right) is better at retaining these color variations while keeping the noise manageable.

Denoise also excels at cleaning up the shadows of low ISO images. We often associate high noise with high ISO settings, but noise can also sometimes be a problem in low ISO images, especially when brightening the shadows. Here’s an example of a high-contrast scene captured at ISO 200 from the Canadian Rockies:

In recent years, we’ve become spoiled by advanced camera sensors, which deliver impressively clean shadows at low ISO settings. However, I made the above picture in 2006 with an older camera that didn’t have all that modern sensor goodness. Let’s examine the brightened trees on the far bank (below left):

Yuck — I’m not a fan of all those color gremlins! Fortunately, applying Denoise at the default Amount of 50 is enough to clean them up (above right) while preserving the tree details.

Best Practices

Here are some tips for getting the most from Denoise.

Batch mode. As with previous Enhance features, you can apply Denoise to several photos at a time by selecting the desired ones in the filmstrip, then clicking the Denoise button. The Enhance dialog will show you a preview for the primary photo, but your chosen options (including the Denoise Amount) will apply to all selected photos. You can also skip the dialog entirely by pressing the Shift modifier key (for Lightroom) or the Option/Alt modifier key (for Camera Raw) before clicking the Denoise button. Using this “headless” option will apply whatever previous Enhance settings you used last time.

Order matters. I recommend applying Denoise early in the workflow, before healing and masking. AI-driven, image-based features such as Content-Aware Remove and Select Subject can be affected by noise, so it’s best to use those features on a clean starting point. If you do run Denoise on an image that already has Content-Aware Remove settings or AI masks, Denoise will automatically update those spots and masks. This is handy, but be aware that the content of those spots and masks may change unexpectedly, so it’s best to review the results carefully.

On a related note, Denoise can sometimes subtly change the overall tonality of the photo, especially by cleaning up the shadows. If your source photo already had major tonal adjustments, such as with Shadows, Clarity, or Dehaze, you may need to revisit those settings after applying Denoise.

Add Grain. Sometimes a denoised photo looks too smooth. Try using the Grain feature in the Effects panel to give your photo a more natural look.

Need for speed. Denoise is by far the most advanced of the three Enhance features and makes very intensive use of the GPU. For best performance, use a GPU with a large amount of memory, ideally at least 8 GB. On macOS, prefer an Apple silicon machine with lots of memory. On Windows, use GPUs with ML acceleration hardware, such as NVIDIA RTX with TensorCores. A faster GPU means faster results.

Optimize your capture. Denoise is very capable, but it’s not an all-powerful genie. Eventually we might be able to type in a text prompt, “Make my ISO 8000000000 picture look like it was taken at ISO 8, thank you, bye!” but we’re not there yet. Until that happy day arrives, optimizing your exposure settings at capture time is still important for best quality. Capturing as much light as you can in the field (“exposing to the right”) maximizes the signal-to-noise ratio. Bracketing exposures and merging them to HDR is also a good approach to getting clean, detailed shadows.

What’s next?

Denoise is our third Enhance feature. We’re proud of what it can do today, but we’re already looking ahead to make it even better. For instance, we have some ideas on how to use additional training data to improve resolution. We’d like to support additional file formats and combine Denoise with Super Resolution. We’re even looking into ways to speed up the workflow by not needing to make a new DNG file. It’s a very exciting time, and you can expect us to continue making big strides forward in AI-powered image editing.

Please let us know what you think about Denoise and how we can improve it! We’d love to see your before/after examples.

Read more about the new features announced today in Lightroom here.