Assessing the Impact of iOS 9 Content Blocking with Adobe Analytics

Given some of the recent developments with iOS 9 and Safari around content blockers (see How to Combat Content Blocking in iOS 9), we want to give you some strategies for leveraging Adobe Analytics to assess the impact this has on your data.

Quick History

On Sept. 16, 2015, Apple released iOS 9 for general use. One part of that release was support for content blockers in the mobile version of Safari (native apps are un-affected). Ad blockers have been around for quite some time and represent a very small percentage of web traffic. However, on mobile devices with slower connections and more limited processors there is a greater incentive for users to adopt content blockers. Several of our customers have expressed concerns that large portions of their traffic will suddenly go dark. We are monitoring how this is affecting our customers, and to-date, we’ve seen very little impact on mobile traffic. We do, however, want to provide you with some strategies for doing the same types of analyses that we are doing. Below are two techniques you can use to determine if you are losing a meaningful amount of traffic to ad blocking.

The Easy Way – Anomaly Detection

The easiest way to look for any sudden changes in iOS traffic from Safari is to use Anomaly Detection in Adobe Analytics. Anomaly Detection uses a Holt-Winters algorithm to predict what metrics should be and what is within the expected bands. Here is how you do it:

1. Create a segment of iOS Safari Users.

2. 
   http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-1.png

3. Open up Anomaly Detection (View All Reports > Site Metrics > Anomaly Detection)

4. Select the Occurrences (essentially hits), Page Views, Visits, and Visitors Metric.

5. Change the training window and view period to 90 days

6. 
   http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-2.png

7. Look for any negative anomalies in these metrics after September 16th.

8. 
   http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-3.png

9. In this data set things look good there is no statistically significant drop in traffic.

As you do this analysis there are a couple of things to keep in mind:

• There are normal variations in traffic. You will notice in the chart above that iOS Safari traffic is down a bit the week after 16th. However, if you look at previous weeks you see a very similar trend.
• A negative anomaly after the 16th means that there ‘could’ be some traffic that is blocked. However, it could be unrelated, so be sure to investigate if there are other possible causes as well. (e.g. bugs in your web site that affect iOS 9 Safari, campaigns that ended, etc.)
• If you run Contribution Analysis on any of these anomalies be aware that iOS and Safari will likely show up in the results set because we are filtering down to just those users. I would only attribute this to ad blocking if Contribution Analysis shows no other significant contributing factors.
• Finally, to be thorough, be sure nothing could be inflating traffic (like a new campaign) that would hide a drop in traffic.

The Manual Way

With this method, Analysis Workspace in Adobe Analytics can help you find a predictor of iOS Safari traffic. You’ll then use that predictor to estimate iOS Safari traffic and compare that estimate to the actual values. Finally, you will see whether the difference is statistically significant.

Finding a Predictor

You want to find a segment that is closely related to iOS traffic but would not be affected by people upgrading to iOS 9 so that you can impute what iOS traffic should be on the site. Here are a few things you might try:

• Traffic from browsers other than Safari

• 
  http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-4.png

• Traffic from iOS devices with a non-Safari browser

• 
  http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-5.png

• Android traffic

• 
  http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-6.png

The best way to do this would be to run a correlation between several metrics and then find the one that most closely correlates to iOS traffic. You can, however, use a trick in Analysis Workspace to quickly pick out which metrics follow the trend most closely—here’s how:

1. Open Analysis Workspace (> Analysis Workspace)

2. Create a new project

3. Change the dates to March 1 – August 30 (roughly six months or the maximum number of days you can display). We want to look at traffic before there were any concerns about ad blocking to find the best predictor.

4. Add in the Day Dimension and drop the segment we created above for iOS Safari on the occurrences metric (occurrences is equivalent to hits or server calls).

5. 
   http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-8.png

6. Add in the additional metrics (Non-Safari, Android and iOS Non-Safari)

7. 
   http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-9.png

8. Add a line visualization and select the normalize option

9. 
   http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-10.png

10. Normalizing puts all the metrics on the same scale. It will allow you to quickly see visually how closely the metrics tie into each other.

11. Deselect all but the first metric. Then view the other metrics one at a time to find the one that most closely follows the traffic line.

12. 
    http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-11.png

13. 
    http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-12.png

14. 
    http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-13.png

15. In this case I would say Android traffic is probably the best predictor, however, feel free to try others or to run a correlation analysis.

Estimating iOS Traffic

The next step is to estimate what iOS traffic should be and compare it against what it was to see if there is a significant difference. To do this you are going to need a few calculated metrics, such as:

• Predicted iOS Safari Traffic from Android

• 
  http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-14.png

• This metric uses the Linear Regression Predicted Y Function. It is one of the advanced functions so be sure to check the advanced checkbox when looking for it. Use the metric you selected above as x to predict iOS safari traffic as y.

• Mean iOS Safari Hits

• 
  http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-15.png

• Upper Bound (2 Standard Deviations)

• 
  http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-16.png

• Lower Bound (2 Standard Deviations). The only thing different is the subtraction operator.

• 
  http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-17.png

Once you have these created you can put them on a chart to get a better indication of changes in iOS 9 traffic. Here’s how:

1. Add a Freeform Table to Analysis Workspace.
2. Add the following metrics:

• • Occurrences for Safari iOS users
  • Predicted iOS Safari Traffic from Android
  • Mean iOS Safari Hits
  • Upper Bound (2 Standard Deviations)
  • Lower Bound (2 Standard Deviations)
  • 
    http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-18.png

3. Set your date range for something relatively long (at least 200 days) so that the mean and predicted values are affected by a drop in iOS traffic (if one exists).

4. Change the row count on the table to 200 and descending (so you see the most recent dates first).
   
   http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-19.png

5. Now add an Area Chart visualization. It should look something like this:
   The straight lines are the mean, the upper and lower bound (at 2 standard deviations). If you go outside of those bands then there is a 95-percent chance that the traffic pattern is not normal. Generally speaking if your traffic is inside of those bands then you shouldn’t be concerned. You can also see that the predicted traffic matches pretty closely the actual traffic, which gives further reassurance that for this data set, there is no measurable effect resulting from iOS 9 ad blocking. If you did have a lot of traffic being blocked you would see the actuals fall well below the predicted iOS traffic and it would fall below your lower bound.
   
   http://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/09/AA-iOS-20.png

6. Save the project. After putting something like this together, you’ll want to hold onto it so you can show your grandkids. You will also want to come back to this view every couple weeks to see if things are changing.

Conclusion

These two methods should give you a good way to assess how impactful the ad blocking is for your business and your audiences. We will continue to share new information as we learn more about how this is behaving in the real world.