5 Data Hurdles in Real-Time Customer Experience Management

Companies are rapidly transforming themselves into digital businesses in a race to deliver the best possible customer experiences. The aim? To build brand loyalty that leads to revenue.

Indeed, we are in the midst of the “customer experience era,” which requires a fundamental shift in the way that businesses compete to win the attention of the modern-day consumer.

Data, of course, is key to delivering the types of experiences that consumers have come to expect. The good news is most companies have no shortage of data at their disposal. However, while the vision and benefits of real-time customer experience management (CXM) are clear, there are still challenges that enterprises face in areas such as data preparation and cleansing, data governance, security, and compliance to consumer privacy regulation. That’s precisely why Adobe is partnering with leading academia to uncover solutions to these challenges, and help companies compete and thrive in the CX era.

To get a better pulse on the data challenges companies face, and identify ways to overcome them, Adobe held its first Academic Data Symposium on October 21, 2019. At the event, thought leaders from five universities (University of Wisconsin-Madison, University of Waterloo, University of Maryland, and University of Massachusetts) shared research with 200 Adobe developers and data scientists. Each presentation generated lively discussions in which professors offered their perspectives on the data challenges inherent in delivering real-time customer experiences along with new ideas on how to solve them.

Adobe’s CEO, Shantanu Narayen delivered the keynote, where he spoke about Adobe’s longstanding commitment to university research. “Investing in deep technology and research with academics is critical to staying at the cutting edge of technology.”

Addressing the challenges of data integration

Enterprises need the ability to understand and use their data to drive insights and action so they can deliver real-time customer experiences. However, heterogeneity in the data remains a significant problem for data scientists. Exploring, profiling and understanding the data, cleansing the data, matching schemas, and extracting structures from it are just a few of the challenges that keep enterprises at all levels from fully leveraging the power in their data.

The University of Wisconsin’s Professor AnHai Doan noted in his presentation that there is a strong temptation for companies to automate data integration processes from end to end. However, he said that doing so would eliminate the benefit of human (a.k.a. data scientists) perspectives and would thus fail in the intended objective. “The [data scientists] need to be in the loop,” he said.

Doan said that when it comes to evaluating the outputs of algorithms, humans are still a necessary part of the process. Instead of trying to eliminate human effort altogether, he said, the focus should be on minimizing it. He further went on to say that this will not be accomplished with a single system.

Doan believes that both academic researchers and the technology industry have important roles to play in the evolution of data integration. “It is our job to identify the pain points and develop different packages or apps to solve these pain points,” he said, adding that approaching these problems as an ecosystem will ensure that the resulting applications will be able to “talk” to each other.

Using machine learning to automate data cleansing

While he agreed with Doan that the human element is important, Professor Ihab Ilyas from the University of Waterloo insisted that there are some problems that can and should be automated end-to-end.

Ilyas said the challenges with machine learning used to be in writing the code to train the models. Today, building and coding AI models is fast and easy with modern tooling.

However, there is still the problem of data cleansing. Clean data is central to getting the right insights for delivering relevant and in-context customer experiences. “Pushing garbage data, and data that has problems, into robust models does not work, especially with structured data,” Ilyas said.

He added that while there is a whole academic ecosystem tackling different aspects of these problems, there are no solutions that offer automated end-to-end data curation infrastructure—until now. In his presentation, Ilyas presented a methodology using machine learning models to automate data cleansing.

Data governance presents many new avenues for research

Data governance is a major concern, and University of Maryland’s Professor Amol Deshpande said that today’s collaborative data analysis workflows present multiple new and difficult challenges in data management and data analysis, including:

All of these above challenges represent important concerns addressed in the Association for Computing Machinery (ACM) US Public Policy Council’s Principles for Algorithmic Transparency and Accountability. Deshpande observed that today’s data science platforms do not support the above principles well due to lack of efficient organization and management of large data sets. These problems include difficulties in tracking lost dependencies, data provenance and parameters, and the inability to compare different pipelines or monitor deployed pipelines for errors and anomalies.

Deshpande believes that academia can help solve these problems and offered the following areas of research which, if explored more deeply, could help the technology industry better uphold the principles of transparency and accountability articulated by the ACM:

Connected data sets

The ability to connect large and varying types of data sets is the life blood of customer experience. However, working with connected data sets still remains a significant challenge for many businesses today.

In his talk, University of Massachusetts’ Professor Marco Serafini said that there is an emerging interest in the use of connected data for machine learning. “We’re seeing more and more work in machine learning where you don’t learn with a flat, tabular data set. [Instead,] you consider relationships and structure as first-class citizens when trying to learn the properties of a data set. Instead of flattening your data, you are using the structure to extract even more knowledge.”

One of the biggest questions when modeling connected data is the choice of the underlying index structure: graph vs. relational. According to Serafini, modeling connected data is less about graphs vs. relational, and more about the difference in workloads. Graph algorithms and graph analytics represent a fundamentally different workload than traditional data management systems.

Serafini said, “There is a need for an evolution.” Rather than reinventing graph database management systems, he said researchers and the technology industry should work toward a convergence of both graph and relational systems for modeling connected data sets.

In his presentation, Serafini provided Livegraph as an example of this convergence – a system to address the challenges of using graph analytics in real-time.

Ontologies and knowledge graphs

University of Maryland’s Professor Louiqa Raschid shares Adobe’s belief in the power of partnerships between academia and industry. At the symposium, Raschid presented her work on building the Business Open Knowledge Network (BOKN), a project dedicated to developing new computational methods for extracting, representing, linking and analyzing data.

While knowledge graphs are not new, most are proprietary to the businesses that build them and are expensive to construct. In contrast, open knowledge networks offer greater access to data and information about businesses, innovation and markets, plus they are freely available. The BOKN is building upon and significantly enhancing the capabilities of existing data resources with:

The BOKN is a stellar example of the vast potential that exists when industry and academia work together. As the BOKN grows, it will offer greater opportunities for academic researchers to develop and test theories with the potential to drive innovation across a wide range of industries to help them deliver better customer experiences.

Adobe invites greater collaboration with academia

At Adobe, we understand the importance of removing the friction at every touchpoint in the customer journey and delighting people with personalized experiences. With rapid advancements in technology, customer expectations are increasingly shifting to real-time demands.

When Adobe’s Chief Technology Officer, Abhay Parasnis, sat down for a fireside chat, during the event, he asked the academics in the room an important question: “What can Adobe do to engage more with researchers in academia?”

One suggestion was to form a committee to answer this question and provide its findings to Adobe’s leadership. Parasnis agreed that this would be a solid step toward greater engagement saying, “We invite conversations with academia about meaningful and material projects and initiatives that intersect with enterprise requirements to meet their customer needs.”

****In the coming months, Adobe will explore the suggestions offered at the symposium to better leverage the great potential for innovation that exists when industry and academia work together.