Changing Dynamics in the Retail Industry
I have always been amazed by the sheer volume of e-commerce data within the Adobe Analytics solution, which is part of the Adobe Experience Cloud. In some sense it has been easy to build our holiday shopping predictions given that our data base covers 80% of the online spend across the top 100 ecommerce companies in the U.S. Over the last five years we have pushed our model to go deeper to explain many facets of a market we are predicting to break the 100-billion-dollar mark between November 1 and December 31 this year.
E-Commerce rocks the world
Unless you’ve been hiding under a rock, you’ve heard about the so-called retail apocalypse stemming from a huge number of physical store closings in the past year. Clearly, ecommerce is rapidly changing the business landscape. The impact of big discount price days is also massively changing consumer buying behavior. For example, when we first started the holiday predictions we recognized that consumers in the UK started spending earlier and ended up spending more. We concluded that America’s obsession with Black Friday was probably reducing the overall holiday spending opportunity for retailers. Two years ago when we expanded our capabilities to dive deep into specific product pricing we realized that the Thanksgiving weekend was in fact the lowest price time to buy nearly every category of products.
We watched the U.S. holiday season become even more volatile and compressed in just a few years. Take a look at how the last three years have seen slow growth during the first couple weeks of November and the last few weeks are growing much more rapidly due to the availability of faster, cheaper shipping as well as click and collect shopping.
What surprised us the most, however, was to see the Black Friday concept exported all over the world to the point where it beats Click Frenzy in Australia and Boxing Day in the UK in just the last two years.
Consumer behavior shifts faster than ever
July is typically a very slow month for commerce so imagine our surprise when we saw nearly 10% growth across the entire ecommerce market on Prime Day, a two year old “sale” day completely fabricated and marketed by Amazon.
Consumers have become conditioned to wait for big discounts and global markets are shifting hundred year old behavioral patterns in just two years. To underscore the impact of these behavioral patterns, consumer’s ability to find the lowest price online is the most likely reason for the lack of inflation in the U.S. economy according to the economic professors at Stanford and the University of Chicago we work closely with.
Given all this dramatic change, as data scientists we realized it was time to rip our model down to the studs and rebuild. If you are familiar with data science or follow other models, you’ve seen this all before. If this is unfamiliar territory for you I’m about to get really nerdy. We want to explain how our model is built and what has changed because we feel so passionate about delivering the very best insights using the most robust analytical capabilities in the world.
Total market size
First, let’s start with the smaller part of the change. Although Adobe Analytics manages a huge percentage of actual ecommerce activity in the U.S. and around the world, it isn’t the entire market. We looked at several different total ecommerce estimates in order to be able to determine total market size and decided to standardize on Forrester Research’s market data. This market estimate is a big bigger than our previous estimate so part of our change is due to the total size we use within our model. The most recent published data available today which is available by subscription only here offers the 2017 total year estimated ecommerce revenue for the U.S. We use Forrester’s data as a base and then apply our own growth calculations within our model based on trillions of actual shopping visits to project the 2017 holiday season spend.
Our Modeling Approach
Second, the larger change is to how we build the model to pick up on changing business environment and consumer behavior. As I mentioned at the onset, we have a ton of data. It is a blessing and a curse. The curse part is that the tools and capacity we had five years ago to crunch that kind of data couldn’t keep up with our desire to report out findings during the same day. At the time, the market was growing pretty consistently aside from a few key days. Our previous model was built with a broader set of constant growth assumptions except for a few large spiky days. This doesn’t mean that every day grew at the same rate last year. It means that we used constant growth assumptions in the model to predict the day’s growth. That was the best choice for the moment.
The assumptions built into our model of the past, however, are no longer valid today. We realized very quickly when actual spending during the first two weeks of November were much lower than we predicted and we made all of the underspending up on the first day of the holiday that we needed to investigate further. That’s why we started rebuilding our model just as soon as last year’s results were finalized. We needed to increase our capabilities to have every single day’s growth estimate be its own independent number. We have spent nearly all of 2017 on our model redesign.
As we refactored everything we discovered an even bigger impact from shoppers moving more and more of their holiday spending to the three biggest online shopping days of the year. We think of those days (Thanksgiving, Black Friday and Cyber Monday) as non-linear, accelerated growth days. The fourth, non-linear accelerated day is Amazon Prime Day and falls outside the holiday season. As with all improvements, however, we must go back and refactor previous years using our new models. This enables us to compare apples-to-apples growth rates. We have restated holiday online sales estimates for 2016 as follows:
For data nerds like the Adobe Digital Insights team, this process has yielded many exciting opportunities to uncover both macro and micro changes to the marketplace and report them in greater detail than ever. We can’t wait to see how retailers – small and large – are adjusting to these changing market dynamics to make sure they don’t become a victim of the so-called retail apocalypse.