How Google’s Expanded Search Encryption Impacts Adobe Analytics

https://blog.adobe.com/media_e9bc57ad8ce3302e488385df25bd64b482bdf83d.gifNear the beginning of this month, Google quietly expanded its encryption of search activity to all organic search results, not just users who are signed into their Google account. Almost two years ago in October 2011, Google announced it would encrypt searches for anyone who was logged into their Google account. By hiding the search queries being performed by individual users, Google positioned this major change as a way of protecting online privacy. The same encryption wasn’t extended to paid search keywords as Google said its advertisers needed the ability to improve the effectiveness of their Adwords campaigns.

This search change in 2011 meant that Google would no longer pass organic search keywords into Adobe Analytics or any other web analytics tool, including even Google Analytics. In aggregate you could still discern how much natural search traffic you were receiving from Google, but you started to lose insight into what the actual search queries were.

Adobe had to create a new line item, Keywords Unavailable, in the Search Keywords report to reconcile this secure-search-related traffic from Google where the search terms were simply not provided. For many organizations, Keywords Unavailable quickly became the top item in their Search Keywords reports (All & Natural) to the ire of many marketers and analysts.

Now with this expansion of encrypted search by Google, you will see no natural search keywords from**** the leading search engine**** and a significant increase in Keywords Unavailable within your Search Keywords reports. Keywords that are associated with paid search ads on Google will remain unaffected and will still appear in your reports. Search Engine Land’s Danny Sullivan obtained the following statement from Google on this change to its search business:

“We want to provide SSL protection to as many users as we can, in as many regions as we can — we added non-signed-in Chrome omnibox searches earlier this year, and more recently other users who aren’t signed in. We’re going to continue expanding our use of SSL in our services because we believe it’s a good thing for users…The motivation here is not to drive the ads side — it’s for our search users.”

Is there a technical workaround? No, not really.

The first question many clients will have to this news is whether there is a technical workaround to this latest Google Search update. The short answer is no—not really. In web analytics, there are often many roads to Rome or, in other words, to the essential data we need. We’ve become accustomed to having various options for collecting data. If one road is blocked, we’ll just take another one. Our natural tendency is to think there must be another way to get this same information.

Before getting into any alternate routes or shortcuts, it’s helpful to understand how web analytics tools capture keyword data from search engines to fully understand the challenge. The keyword data is typically visible in the HTTP referrer field of the page request. For an organic search for “analytics hero” on Google.com, the referrer field in the HTTP request used to look like this:

http://www.google.com/url?sa=t&rct=j&q=analytics%20hero&esrc=s&source=web…

For comparison purposes, here’s what the referrer field still looks like for both Bing and Yahoo today:

http://www.bing.com/search?q=analytics+hero&qs=n&form=QBLH…
http://search.yahoo.com/search;_ylt=AlwlrXz6Kg5n2hhtmGeKI4ybvZx4?p=anaytics+hero…

The keyword is embedded within the referrer’s URL and can be easily extracted for keyword reporting. If you were to look at the referrer field today for all organic searches from Google, this is what you’ll see:

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web…

Ugggh. You’ll notice the q= key-value pair is missing the value for search terms. As a result, Adobe Analytics has no idea what keywords were used prior to someone arriving on your site. There is no way to retrieve these terms if they are being hidden or encrypted by the search engine before visitors land on your website. Unfortunately, there is no backdoor, and only Google can provide this data.

Through Google’s Webmaster Tools you can obtain an aggregated list of the top 2,000 search queries that drove traffic to your site for the past 90 days. This might sound like a satisfactory workaround until you realize it is simply a raw count of the most popular search keywords with no connection to your engagement or conversion metrics (or visitors). While you might be able to extrapolate insights from your paid search keywords, it’s going to be much more challenging to optimize your SEO efforts for Google without meaningful data on specific natural search keywords.

What are the repercussions?

Unfortunately, when the dominant search engine decides to change the keyword information it shares, every website—small and large—is impacted. Here’s a quick summary of what you’ll see in Adobe Analytics:

An increase in Keywords Unavailable in both Search Keywords reports – All & Natural.
Natural Search Keywords report will only feature keywords from other search engines that still provide organic keywords such as Bing and Yahoo Search.
Paid Search Keywords report will be unaffected.
Segmentation and breakdowns by natural search keywords will be less effective as they only represent non-Google search traffic.
In the Marketing Channels reports, the details breakdown for the Natural Search channel will be less useful due to having a greatly reduced amount of keyword data.

Why do you still see organic keywords from Google?

Most of Google’s search changes roll out gradually across its infrastructure. We saw this with the original change to secure search. If you trend Keyword Unavailable over the last month or so you’ll probably notice a gradual increase like I did. For my website in the graph below, you’ll see the change started to appear in August. We don’t know when Google will reach 100% rollout, but it is going to happen sooner rather than later.

https://blog.adobe.com/media_8890962206120ddf0383cdcf3cb521509e065971.gif Is there anything you can do about it? Yes.

When the going gets tough, action heroes don’t give up–they get moving (sorry, couldn’t resist). While Google has limited what you can do from a technology perspective, you still have several analysis techniques at your disposal to soften the blow from Google’s search encryption. Here are six ideas for your next keyword rescue mission:

Even though your other search sources may only represent a small portion of organic search volume, they can still be helpful from a directional perspective. The data will continue to show you which keywords convert and which don’t. Optimizing based on the organic traffic from Bing or Yahoo should help you across the board.
If you haven’t already been using the Webmaster Tools’ Search Queries report, start experimenting with it today to see what interesting patterns and insights it can yield. It rarely matched up with search keyword reporting before the change, but at least it could provide some insight into significant shifts happening with your keywords.
From your landing pages, you might be able to infer the intent of the visit or the type of organic keywords that brought someone to the site, including branded and non-branded traffic. More than ever it is critical that you have separate landing pages for paid search traffic so that it doesn’t obscure your insights into your organic traffic.
You can create a segment for your natural search visitors, and evaluate what entry pages they’re using and where they’re going. Similarly, you might be able to deduce their intent and keyword usage.
Pay attention to patterns and spikes within your internal search terms. People who are unable to find what they’re looking for will turn to search for help, which may happen more frequently as the content can’t be as readily tailored to meet the needs of Google’s organic search visitors.
Evaluate site navigation patterns once they’re on your site. A simple technique called destination page analysis can help determine where visitors are migrating after they’ve entered your site.

While none of these techniques will replace your lost keyword data, I’m hopeful some of these ideas will come in handy. Unfortunately, with a topic like this, I hate to be the harbinger of disappointing news, especially when there’s not a simple or quick fix. However, it’s important for you to be aware of this change and understand how it impacts your reporting and analysis in Adobe Analytics. If you have further concerns that I have not addressed in this article, please let me know. If you have suggestions based on how your company is handling this change, please share them with your peers and allies within the Adobe Analytics community. In a future blog post, I’ll share my own personal perspective on what drove this key shift in the search marketing space.

Follow me on Twitter @analyticshero.