Shedding Light on Dark Social with Adobe Analytics

Ever since Alexis Madrigal’s seminal piece on the topic in The Atlantic two years ago, analysts and marketers—especially at media and entertainment companies—have been vexed by the “Dark Social” problem. In digital analytics terms, Dark Social represents that portion of traffic or conversion that should be attributed to social sites like Facebook and Twitter, but aren’t. The reasons that analytics solutions haven’t been attributing traffic to the social channel correctly vary, but they all boil down to whether or not the referrer gets passed across a link click and is available on a landing page.

As an illustration, consider the Facebook app on your mobile phone. When you click a link in Facebook—say a friend tweets out a link to an article you find interesting—even though you are likely staying within the app itself, you are taken to a web browser instance and you load a web page, not part of the Facebook platform. The problem is that Facebook does not send a referrer to the browser, and if a referring site does not identify itself on your landing page, your analytics solution cannot capture that referrer using traditional, automatic means. All traffic that you generate in this scenario will therefore appear to be “direct” or “typed/bookmarked.” Thus, even though the page view, the visit, the visitor, and all downstream activity is recorded in your analytics solution, these metrics are not associated correctly with Facebook.

The result of Dark Social, therefore, is that the “direct” or “typed/bookmarked” channels receive credit for a lot of traffic (and conversion!) that they shouldn’t, while your marketing efforts and the virality of your content receive far less credit than they should. As an analyst, you should be annoyed! This phenomenon has been skewing your acquisition data for too long, muddying the picture you have of your customer/reader/viewer journey.

What Is a Marketer to Do?

Three years ago, Adobe Analytics introduced a feature called Processing Rules. Processing Rules allow you to set and edit values based on the contents of the data Adobe is collecting from your web site or mobile app. To open up a ton of great possibilities, it even lets you use data points not normally available within standard reporting as inputs to your rules. This is how you can begin solving the Dark Social problem today.

Of particular interest to us is the user-agent string, which is used to identify the browser version. While Facebook and other apps do not send a referrer to the landing page as part of the in-app browsing experience, several of them do identify themselves as a special instance of the web browser, using the user-agent string. Here are two example user-agent strings, both from the Facebook mobile app, one from iPhone and the other from an Android device:

iPhone
Mozilla/5.0 (iPhone; U; CPU iPhone OS 5_1_1 like Mac OS X; ru_RU) AppleWebKit (KHTML, like Gecko) Mobile [FBAN/FBForIPhone;FBAV/4.1;FBBV/4100.0;FBDV/iPhone3,1;FBMD/iPhone;FBSN/iPhone OS;FBSV/5.1.1;FBSS/2; tablet;FBLC/en_US]

Android
Mozilla/5.0 (Linux; Android 4.4.2; SCH-I535 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36 [FBAN/FB4A;FBAV/22.0.0.15.13;]

Notice the very ends of these strings. All of those four-character strings beginning with “FB” aren’t part of the normal Safari or Chrome user-agent string. This is how we are going to identify that a visitor is browsing your site from within the Facebook app. This is how we are going to account for (at least a major chunk of) Dark Social.

It’s important to note that, to keep this post simple, I’m going to focus on identifying Dark Social from the Facebook mobile app. Facebook on web does not face this problem nearly as severely, because the referrer is passed successfully in most cases. There are other applications, both on desktop (think instant messaging clients) and on mobile/tablet devices, that similarly spawn a new browser window and therefore do not pass a referrer. These are identifiable to varying degrees. In a mobile app, you can post a link to whatsmyuseragent.com and then click through that link to see whether the app identifies itself by user-agent string. Another good tactic is to check for the presence of a referrer by going to a landing page and replacing the URL in the address bar with javascript:alert(document.referrer) and hitting Enter.

Building Your Processing Rules

If you’ve never worked with Processing Rules in Adobe Analytics, check out blog posts here and here, as well as documentation and certification information here. You’ll need to pass a free certification before gaining access to Processing Rules. Once you’re certified, you can proceed with the steps below.

The processing rule we are going to create is actually really simple. User-agent is one of the fields that I can use in my conditional statement, and then I am going to enter _each identifier _that may indicate a dark social click-through on a new line. When you’re done, the rule should look like this:

The following aspects of the rule are key:

• Make sure that the operator for the condition is “contains,” not equals. You want the rule to look for aspects of the user-agent string, but not the entire user-agent string.
• You’ll want to set a value into a spare eVar. If you are out of eVars, you can use a prop. Below, we’ll use the value of this variable to set a rule in Marketing Channels to combine dark social with other social data. You may want to do both.
• As a bonus, you can create two rules—one for iOS and one for Android. “FB4A;” is unique to the Android user-agent, so you can create a first rule which looks for that string, then create an iOS rule that looks for the other strings.
• For the custom value you are setting into the variable, you can be as granular or specific as you want. I chose a value of “Facebook Mobile App” in the screen shot above, but I could have broken that down by operating system (see previous bullet) or I could group multiple apps into a single “Dark Social” value. It depends how I want to report on Dark Social.
• Another thing that you may want to do if you will be using Dark Social in the Marketing Channels tool (below) is to additionally pass referring domains into the same eVar/prop as your Dark Social identifier. This should come in a rule listed after the Dark Social rule.

Hammering it Home with Marketing Channels

Once your processing rule is set up, the next step is to combine it with your social acquisition data in the Marketing Channels tool so that Marketing Channels will attribute traffic and conversion to Dark Social. (If you haven’t already set up the Marketing Channels tool, here’s some documentation that you can use to get started.)

You probably already have a Marketing Channels rule for the Social channel. If you do, all you will need to do is add a rule for Dark Social to cause the rule to trigger if the eVar (or prop) that you set in the Processing Rules setup is present.

Here, my eVar was named “Dark Social,” so I am telling Marketing Channels to count a Social referral whenever there is a value in that eVar. Optionally, if you are passing all referring domains into the eVar/prop that you are using for Dark Social, you will want to change the last drop-down menu in the rule (“Set the channel’s value”) to the eVar or prop in question. This ensures that you can drill into the Social category to get the next level of granularity, so that you can compare Dark Social to other social media acquisition sources.

What you should see, once these rules are set up—and provided you get a reasonable amount of traffic from Dark Social—is an increase in traffic and conversion attributed to the Social Networks channel, and a decrease attributed to the Direct channel. While these changes will not be retroactive, they will be all set moving forward. It is also worth noting that you can use the prop (Custom Traffic variable) recommended above in the Real-Time report to begin reporting on Dark Social against your other referring domains in real-time.

Next Steps

Dark Social continues to be a challenging issue for analysts, marketers, and content owners. What I have proposed here is an initial solution, but Adobe is far from done thinking about how to provide you with better insight into the virality of your content. There is much more that we want to do to provide you with better, more automated insight around Dark Social. For now, I hope that this gets you over the hump to help you answer some of the questions you’ve been getting from your colleagues and customers, both internal and external.