Why Unique Visitors can be so different: Demystifying Web Analytics and Panel-Based Audience Measurement Services

Do you use multiple measurement systems to get a handle on your website traffic? For example, a online business optimization or web analytics platform like Omniture SiteCatalyst alongside a panel-based audience measurement service like Comscore Media Metrix, Hitwise, or Nielsen NetRatings? If so, you’ve undoubtedly wondered why the metrics reported by these systems often don’t line up. In particular, why are unique visitors often different? How can one system report 5 million unique visitors per month, and the other system report 2 million?!

These are among the most frequently asked questions I hear from media and content customers. But this issue extends well beyond media companies. As it turns out, many retailers, business to business, and travel websites also use web analytics and audience measurement services side by side. And wherever these two platforms co-exist, there are always questions as to why the numbers do not line up.

So in this new blog series, I wanted to share my own thoughts on the matter. And perhaps most importantly, I wanted to offer some guidelines on when and why you should use one set of figures vs. another.

COMPARING APPLES TO ORANGES
To begin with, it’s critical to understand that online business optimization or web analytics (i.e. Omniture SiteCatalyst) and panel-based audience measurement (i.e. Comscore Media Metrix, Hitwise, Nielsen) use fundamentally different measurement methodologies. Even though they use similar industry terms, such as unique visitors, you’re really comparing apples to oranges if you try to compare both sets of numbers. Some of you already know this, but many people actually do not realize how fundamentally different they are.

WEB ANALYTICS AND UNIQUE VISITORS: DEMYSTIFIED
To demystify these methodologies, let’s start with web analytics. In general, web analytics or business optimization platforms like Omniture SiteCatalyst measure online activity at the visitor level for roughly 100% of your unique visitors. However, as I’ve highlighted in previous posts, unique visitors is a fairly loose term in web analytics. As it relates to this discussion, one of the most fundamental differences is how web analytics packages typically measure unique visitors.

Specifically, web analytics platforms typically define a unique visitor as: a unique cookie set on a unique computer and a unique browser that accesses your website.

So there are three key pieces there – unique cookie, unique computer, and unique browser. If you change any one of these, you get a new “unique visitor”. For example, if you use Internet Explorer to visit one website, then switch to Firefox – this counts as two unique visitors. Similarly, visit a website from work and then from home – two unique visitors. Your kids visit a website from your computer, you visit the same website from your computer – that’s one unique visitor, even though it should arguably be two. This is really important to understand, and one of the main reasons you need to be cautious in using unique visitors as a strategic metric.

AUDIENCE MEASUREMENT AND UNIQUE VISITORS: DEMYSTIFIED
Now let’s look at audience measurement and how those services typically measure unique visitors. Audience measurement services generally leverage a sampling methodology that measures *some* of your website visitors and then projects the total number of unique visitors from that sample. Depending on which service you use, this sample can vary dramatically. Recent figures I’ve seen for one of the major audience measurement services suggests their US sample is about 120,000 panelists. These panelists are recruited thru differing strategies including random digital dialing (RDD) and/or web-based offers, and the services work very hard to ensure that these samples are representative of the total Internet population. In turn, these services project the total US internet population based on these panelists. For example, as of May 2006, the audience measurement service using 120,000 US panelists projected the active US Internet population at 172 million. This means, on average, that 1 panelist represents 1,400 unique visitors.

Unlike web analytics platforms, audience measurement services go to great lengths to measure truly unique visitors. Participants on their panels are required to use separate log-ins so that all activity can be tied to an actual individual. Furthermore, these individuals provide a fair amount of demographic and attitudinal data about themselves, so not only can you view these unique visitors as distinct people, but you can also dive quite deep into who they are and what motivates them.

For all practical purposes, audience measurement platforms could define a unique visitor as: an individual person participating in a panel that accesses your website.

In this case, the key pieces here are really that the individual is participating on the panel, and that the panel is representative of your web site audience. So referring back to the initial discussion, if 1 of these unique visitors visits your website, the audience measurement services should report 1,400 unique visitors to your website.

But in truth, this isn’t necessarily the case. So let’s take a look at some reasons why.

NOT ALL PANELISTS ARE CREATED EQUAL
If you dig into audience measurement services, you’ll notice that their “panel” is actually several panels rolled up into one. The 3 major panel types are at-home, at-work, and university. These correspond to the actual physical location of the end-user, who can presumably access the Internet from one or many of these locations. Now, this may seem nothing more than a nuisance; but in reality it’s a major issue that audience measurement services struggle with. Why? Because while enticing at-home users with free virus protection software tends to be fairly effective, at-work users don’t really care. And even if they did care, few corporations are willing to participate in these services. So recruiting an at-work panel is *significantly more difficult and expensive* than recruiting an at-home panel. And most of those at-work panelists are confined to smaller firms with less than 100 employees. Tapping into medium sized businesses with over 500 people is next to impossible; and the Fortune 2000 is all but a pipe dream.

To illustrate this point, one audience measurement service I recently worked with has a 10:1 ratio of at-home to at-work users. That’s a staggering difference. And what’s more – these audience measurement services report that the at-work audience is much more active than the home audience. For instance, in June 2006, Nielsen NetRatings reported that at-work internet users averaged 17 visits per person and visited 39 domains. This compares to the at-home users that average 10 visits per person and visited just 25 domains. But perhaps most important is the time spent per person. The at-work internet user spent roughly 19 hours online in the average week. The at-home internet user, by contrast, spent just under 9 hours per week. More than 50% less!

So the at-work internet users represent a critical population of users; yet most audience measurement services have a very difficult time accessing them. This is where the proprietary audience measurement methodologies really kick in. To compensate for this massive bias in the data, the audience measurement firms “weight” the data. In other words, they give more credit to users on the panel that are under-represented and give less weight to those that are over-represented. This weighting is based on their assessment of what the “normal” internet population looks like, which they derive thru proprietary means but random digital dialing is certain one frequently used approach.

With audience measurement services, not all panelists are created equal and therefore not all unique visitors are created equal. As we’ve seen, there can be huge differences depending on how under-represented a particular panel demographic may be such as at-work professionals in Fortune 2000 companies.

QUANTIFYING THE AUDIENCE MEASUREMENT BIAS
From a measurement perspective, this starts to impact sites that are concentrated among particular demographics and deviate from the broad internet population. For example, if you are a business to business technology website, you’ll probably draw very few panelists to your site because this target demographic is among the hardest to recruit for the panel. So the audience measurement services must significantly weight these few panelists to your site and instead of the earlier 1 to 1,400 unique visitor ratio we discussed, it may be more like 1 to 10,000. That’s right – 1 to 10,000 – if not more.

Now, to illustrate the impact of that bias, let’s say these audience measurement services report you had 1 million unique at-work visitors in June 2006. Assuming they all more or less meet the above demographic, that’s really only 100 people that visited your website in June 2006 (with a 1 to 10,000 ratio). Of course, it’s unlikely your website attracts one distinct demographic – but that’s beside the point, because most demographics you do attract will be underrepresented as it is. So while your web analytics package is measuring roughly 100% of unique cookied browsers on a unique PC, your audience measurement service is measuring just 100 people at-work. So which is right? Which should you use? I’ll get to that in a bit…let’s dive a bit deeper first.

This panel bias also means that during abnormal periods of internet activity, the audience measurement service will likely report even more volatile unique visitor figures. These abnormal periods may include any strategic campaigns you run, such as search engine marketing, email blasts, holiday promotions, etc. It may also include seasonal spikes such as political events (i.e. presidential elections), sports events (i.e. World Cup, World Series, Football, Olympics), financial events (i.e. tax season), or just breaking news stories (i.e. Hurricane Katrina, Iraq War, etc).

During these periods, you may see dramatic spikes in traffic reported by your web analytics platform, your commerce platform, your ad server, your server web logs, etc. – but the increase reported by audience measurement services will be a fraction of those. This has often been a substantial point of frustration for internet marketers, because it’s arguably an inaccurate view of performance. In fact, I’ve worked with some customers that show a 40% increase in web site traffic during campaign periods, and have their audience measurement services actually report a decrease in overall traffic!

BIGGER SITES ARE STILL AT RISK
There are some schools of thought that suggest the larger websites are fairly immune to this bias because they attract a fairly representative population of users. The argument goes that the more representative your population, the more closely the panel will tie to your demographic mix.

While this is a plausible theory, I find it suspect at best. My main issue is that 30-35% of the internet population accesses the Internet from work. As I mentioned earlier, this extends to the time spent on the web as well – with at-work users spending over 2x as much time online as at-home users. What’s more, the at-work audience has come to represent a significant portion of online commerce spend and this only continues to increase. Comscore has reported that the at-work audience contributes 60% of online spend. Yet, this coveted demographic often represents 10% or less of a given audience measurement panel – and even that 10% is skewed to small business and mom-and-pop shops, rather than corporate America at large. So when measuring unique visitors, most websites will be adversely impacted by the inherent bias of audience measurement services – irrespective of their size.

LET’S NOT FORGET INTERNATIONAL VISITORS
I lastly wanted to touch on International visitors. Most audience measurement services grew out of the United States, though there have been a few International ones that achieved a healthy foothold in the market. In any case, International panels tend to be much smaller and fragmented than their US counterparts. To underscore that point, many audience measurement firms do not even report at-work activity for International visitors. As such, many websites that rely on these services to measure unique visitor traffic are missing a meaningful portion of their audience altogether. It’s not under-represented; it’s simply not represented at all. Ironically, this issue often becomes more detrimental as your website and business grows.

For example, one company I have worked for derives over 15% of their revenues from Japan. The portion is even greater for their online revenues specifically. Japan’s internet population has been estimated at between 80-90 million users. At least half of these users are accessing from work, although a few older studies suggested as much as 75% of Japan’s internet population was at-work. In either case, most audience measurement services today do not currently offer at-work panels for Japan. In other words, using such services to gauge overall web success would effectively cut your audience in half. Of course, as your presence in this market grows, the gap only increases.

SO WHAT DOES THIS MEAN FOR YOU?
We covered a lot of ground in this first blog posting, so take some time to soak it in and ponder what you’ve just read. If you use web analytics or business optimization platforms alongside panel-based audience measurement services, you’re probably a bit frustrated by all of this. Why can’t I just have one right answer, instead of two answers that are arguably both wrong? Why doesn’t someone offer the best of both worlds? Hasn’t someone solved this problem yet?

Indeed, these are all fair questions. The key is that each respective measurement approach is actually quite valuable – if you know how to apply them. Web analytics or business optimization can be leveraged to drive substantial gains in your top and bottom-line. Audience measurement services can also be leverage to inform significant strategic decisions. But to get there, you need to first stop comparing numbers from each system. Once you’ve gotten comfortable with that concept, the possibilities for business improvement are virtually limitless.

In my next blog posting, I’ll share my thoughts on how you might consider using both measurement platforms to achieve greater business success. Until then, I welcome your feedback and comments!