Advance Your Bot Filtering Powers
by Eric Matisoff
posted on 08-03-2017
Bots have been driving added frustration amongst companies lately, and across more teams. IT puts up with the additional server load, Operations often worries about proprietary data that bots are collecting, and Marketing deals with increased data noise. The good news for marketers using Adobe Analytics is that we have enabled our customers with several options for dealing with this influx of bot traffic that may be significantly affecting data quality. As described in this blog post, our built-in IAB Filtering plus manual bot rules are ideal for staying on top of the most recently identified bots and removing them from your report suites. Unfortunately, bots are advancing quickly and it’s become somewhat of an arms race to get rid of them.
The good news is that we have several powerful features that when combined properly, can help drive the removal of these enemies of data quality. Those features are: Marketing Cloud ID Service, Segmentation, Data Warehouse, Customer Attributes, and Virtual Report Suites.
Now let’s unplug some bots!
1 – Pass your visitors’ Marketing Cloud ID into a new declared ID:
To start, you’ll want to create a new declared ID in the Profiles & Audiences core service. You’ll need to pass your visitor’s Marketing Cloud ID into this new declared ID, which can be done quickly and easily with Adobe DTM. I’ve used the name “MCID” for my declared ID.
Here’s a screenshot of how this ID can be captured via Data Element. Be sure to populated your Adobe MCOrg ID into the Data Element correctly.
Once this Data Element is set up, follow the instructions on this post to pass declared ID’s into the MCID Tool in DTM.
2 – Use segmentation to identify pesky bots
Now that you have your visitor’s MCID passed into a declared ID, it’s time to use segmentation in Analysis Workspace to identify visitors that are acting bot-like. Bots are often defined by their behavior: single access visits, unusual user-agents, unknown device/browser information, no referrers, new visitors, unusual landing pages, etc. Use the powers of Workspace drilldowns and segmentation to identify the bots that have snuck through the IAB filtering and your report suite bot rules. For example, here’s a screenshot of a segment that I am using:
3 – Export all the MCID’s from the segment via Data Warehouse
Now that you’ve identified the bots using segments, the next step is to leverage Data Warehouse to extract all the Marketing Cloud ID’s associated with this segment. Here’s how I set up my Data Warehouse report:
Remember to use Marketing Cloud Visitor ID as your dimension and apply the Bots segment.
4 – Pass this list back to Adobe as a Customer Attribute
Once the Data Warehouse report arrives, you’ll have a list of MCID’s that need to be filtered from historical data. Copy and paste these MCID’s into a blank CSV file with just two columns, MCID and Bot Flag:
Make sure the first column header matches the name you gave to the new declared ID above.
Use this CSV as your Customer Attribute import file, then subscribe your report suite(s) to the Customer Attribute as described in this previously mentioned post.
5 – Create a segment that leverages the new Customer Attribute
Once your data set has been processed and integrated into Analysis Workspace, create one more segment that leverages your new “Bot Flag” customer attribute dimension:
6 – Use this segment as your Virtual Report Suite filter
Finally, you’ll want to create a Virtual Report Suite (or edit one you’re already using) that leverages this segment to filter out the identified bots:
This newly-segmented Virtual Report Suite will now be a significantly cleaner set of data with the identified bots completely removed.
7 – Repeat steps 2, 3, and 4 regularly
I recommend setting at least a monthly reminder to identify and filter new bots, perhaps prior to regularly scheduled analysis.
Enjoy your improved data accuracy! If you have questions, comments, or ideas, don’t hesitate to reach out via the Adobe Analytics Community Forum.