The Future of Retailing

Big Data and Analytics in Retailing

Venky Shankar


Big Data, Analytics, Retailing, Automation, Data Science, AI, Machine Learning

download pdf

What is the big deal about big data?
Big data are taking center stage for decision-making in many organizations, especially retailers. The McKinsey Global Institute has predicted that retailers embracing big data can increase their operating margin by more than 60 %. There is an explosion in data availability and collection. Business data double every 1.2 years. Specifically, customer-related data are growing by leaps and bounds. These data include online browsing data, social media data, mobile usage data, purchase data, customer satisfaction data and the like. For example, a retailer like Walmart collects data on about 1 million transactions per hour, contributing to 2.5 terabytes of data. Furthermore, with the runaway growth of the Internet of Things (IoT), more data are continuously collected from sensors in multiple devices such as smart watches, smart speakers and other wearables that are connected to the Internet. These data require massively parallel software running on thousands of computer servers often in a cloud-based environment. By some estimates, in 2020, one third of the data will be processed through the cloud, yielding 35 zettabytes (35 x 1021 bytes) of data, which may be spread across about half a million data centers across the world.

How can retailers better understand and leverage big data?
Retailers like Amazon are constantly collecting, curating, and analyzing data, and making critical decisions. Their decisions, in turn, fuel customer interactions with retailers with more data which are again recorded, processed and analyzed for further decisions. Many such decisions are made in real time. Thus, the cycle of constant data collection, analysis, decision, and further data collection keeps escalating with larger volumes of data. By some estimates, the market for big data is expected to be $56 billion in 2020. The framework in Figure 1 can help to understand big data analytics and its impact on retailing.

In this framework, customer data on attitudes and behavior across channels, touchpoints, devices and platforms are constantly recorded and collected. These data are integrated from multiple sources and stored or warehoused, often in a cloud-based environment. Statistical, econometric and data science models are developed for enabling appropriate decisions. Computer algorithms and programs are created for these models. A class of models, called machine learning-based models, are particularly useful for learning from the data and making predictive decisions. Many decisions, especially continuous and real-time decisions, are automated. These machine learning models form the backbone for the generation and development of AI-assisted decisions. In many cases, such decisions are automated using systems such as chatbots and robots. For example, chatbots assist in customer service and robots help in warehouse and retail store automation.

Hindsight, insight and foresight from data analysis
Data science models are at the heart of this framework. These models can be classified as descriptive, predictive and prescriptive models. Descriptive models primarily capture past behavior and data. The outputs from these models can be viewed as hindsight. Predictive models predominantly offer forecasts of focal outcomes. These models typically offer insight for retailers for decisions. Prescriptive models focus on providing normative decision recommendations. These models can be thought of as offering foresight. They incorporate optimization of focal decision variables. In the retail pricing context, a descriptive model could be a demand model of how customers responded to past price changes. A predictive model could be one that predicts future sales response to price changes. A prescriptive model is one that offers optimal price recommendations to retail managers. Because a large retailer deals with several thousands of items with millions of customers and perhaps billions of transactions, such a pricing problem is truly a big data problem.

How retailers can benefit from big data
Of special interest are issues such as omnichannel shopping behavior, resource allocation across channels, the effects of the mobile channel and mobile apps on shopper behavior, retailer pricing (in particular, dynamic pricing), data privacy and security. Research on these issues reveals several interesting insights on which retailers can build.

Omnichannel shopping behavior
Multichannel and omnichannel shoppers are typically more valuable than single channel shoppers. However, for certain categories, single channel shoppers may be more valuable than multichannel shoppers. A mobile channel typically enhances overall purchases. In particular, mobile apps have interesting effects on shopping behavior. They lead to greater frequency, quantity and monetary value of purchases in both online and offline channels, but also result in greater product returns. However, overall, mobile apps lead to greater monetary value of purchases net of returns. By the same token, a failure in a mobile app can lead to decreases in the frequency, quantity, and monetary value of purchases in offline channels. Also, marketing efforts in one channel tend to have cross-channel effects. Therefore, retailers need to carefully analyze big data and leverage the findings for improved decisions.   

Personalized recommendations and offers
Amazon, for example, uses big data about its customers, including over 100 million Amazon Prime customers, mainly for predicting customer purchases, making personalized recommendations of offerings and optimizing supply chains. Retailers such as Kroger and Safeway use big data to offer weekly promotions. Many of these retailers’ models are based on machine learning. The larger the customer base and the number of interactions and transactions, the bigger the training data. The more the training data, the better the algorithms’ learning from the data, which in turn translates into more accurate predictions of future customer behavior.  

Customer relationship management
An integrated database is critical to business success, so many retailers are investing in creating such databases. Retailers are also focused on using analytics to identify new sources of revenues to improve topline growth, as well as identify and implement profitable customer relationship management (CRM) strategies. To fully leverage big data in today’s retailing environment, CRM strategies must be location specific, time specific and channel specific in addition to being customer specific.

Box 1: The data privacy challenge
Many highly publicized data breaches of retailers have heightened consumers’ concerns about privacy. The General Data Protection Regulation (GDPR) is a sweeping set of new rules developed by the European Union to protect consumers in Europe. Noncompliant retailers can face fines up to 4 % of company revenues or 20 million Euros, whichever is greater.

Unfortunately, many retailers haven’t made any meaningful changes to their data collection and use to the point of noncompliance. Retailers will have to explicitly seek consumer permission for data gathering and processing. Consumers expect to be able to control how their personally identifiable information is used. Therefore, retailers must be sophisticated in their use of technology and scrupulously follow security procedures. However, privacy concerns should not be a reason for failure to embrace the promise of data-driven decisions. Retailers that leverage data with business analytics will be able to identify profitable products and services, as well as target customers effectively throughout their purchase journeys.

The GDPR may significantly alter retail marketing. It will make behavioral data collection more challenging. To comply with the new rules and still be effective, marketers may have to adopt practices such as contextual online advertising. This means rather than use a consumer’s profile to target an ad online, marketers may have to serve an ad based on the content of the article, blog or webpage that a consumer is viewing in real time. Behavioral data collection through the use of cookies, geofencing and app monitoring will not go away entirely. Retailers will have to be more transparent, better secure consumer data and be more creative in the collection, processing and use of data.

The future of big data in retailing
Leading-edge practitioners of big data in retailing such as Amazon and Alibaba are developing more advanced machine learning models to continue their lead over rivals. Such models are driven by deep learning algorithms. Most deep learning models are based on neural networks. These deep learning models form the engine for smart AI systems. AI is pervading all consumer tools, ranging from Siri to Alexa and Gmail. The future will be dominated by AI-assisted customer behavior and AI-based managerial decisions. Automation will continue to grow and replace or reshape jobs. By some estimates, AI could lead to the displacement of about one third of the jobs in the retailing industry. Whatever happens in the future, one thing is clear: Big data and analytics will be the bedrock of smart retailing in the future.


Venkatesh (Venky) Shankar, Professor of Marketing and Coleman Chair in Marketing, and Director of Research, Center for Retailing Studies, Mays Business School, Texas A&M University, USA, venky@venkyshankar.com

Further Reading

Bolton, Ruth and Shankar, Venkatesh (2018): “Emerging Retailer Pricing Trends and Practices,” in Handbook of Research in Retailing, Katrijn Gielens and Els Gijsbrechts eds., Edward Elgar, MA, 104–131.

Kushwaha, Tarun and Shankar, Venkatesh (2013): “Are Multichannel Customers Really Valuable? The Moderating Effects of Product Category Characteristics”, Journal of Marketing, Vol. 77 (4), 67–85.

Narang, Unnati and Shankar, Venkatesh (2018): “Mobile Marketing 2.0: State of the Art and Research Agenda”, Review of Marketing Research, forthcoming.

Narang, Unnati and Shankar, Venkatesh (2018): “The Effects of Mobile Apps on Shopper Purchases and Product Returns”, Working Paper.