download pdf

Catering to anonymous customers
In a supermarket, an exemplary shopper, let it be Paul, takes a shopping cart and makes his way through the aisles. From the perspective of the supermarket, he is yet another anonymous customer. Paul takes products from the shelves, looks at the packages, compares information and decides to put some products in the cart and others back on their shelves. At some point, information regarding allergens puzzles Paul and he finally decides to abandon a product even though the item was on his shopping list. After waiting some time in line for checkout and paying cash, Paul returns the shopping cart, which now waits for the next unknown customer to move it around. Retailers know very little about consumers like Paul. Although security cameras may have filmed him several times, the tapes are probably deleted after a while and no other usable personal data will be retrieved or stored – data that may help the retailer improve the shopping experience, retain customers and ultimately increase profits.

For online retailers, the situation is different
Visitors of an online retailer’s website leave digital traces. Every click and each interaction on the website generate information about a customer. During registration or for a purchase, their names, addresses and often more information are revealed. Digital retailers leverage big data and smart algorithms to reveal consumers’ wants and needs, predict shopping behavior, recommend products, and optimize inventories to minimize storage cost and delivery times (see Figure 1).

Recently, digital players like WeChat in China and Amazon GO in the USA und Europe started bridging online and offline: They establish automated, self-service or even checkout-free supermarkets that offer unprecedented levels of shopping convenience and generate additional data about customers.

Technologies that can help brick-and-mortar stores catch up
Market incumbents need not watch this development in passive awe. Many technologies are already available that help brick-and-mortar stores gather more valuable information about their customers, allowing them to improve shopper experience, retain customers, and ultimately increase profits. If used in a smart, transparent and non-offending way, the technologies described below can make a physical store almost as data-rich as a website (see Figure 2).

In-store cameras and sensors for tracking the customer journey
Insights from tracking the shopper path in a store can be used for many purposes, from optimizing the store layout over deciding on secondary placement locations to time-of-day-dependent adjustments of offers.For a proof-of-concept, we developed a typology of shopping trips based on path-tracking data collected over a one-year period in a German supermarket, using UWB (ultra-wide band) antennas that monitor battery-operated tags implemented in shopping carts and baskets. Based on characteristics like distance covered on a shopping trip, speed, and the proportion of trips spent in specific areas of the store, we identified eight different shopping trip types. For instance, “unstructured refills” are characterized by a long distance, whereas, “single purpose” and “last-minute trips” by a high speed. Combined with survey and sales data, more detailed behavioral differences between segments can be carved out. Such data allows targeted recommendations for specific segments, such as reminders for frequently forgotten items for unstructured refills, optimized category management and bundled secondary placements for single purpose trips, and special product aisles for last-minute shoppers.

Observing customer decisions: Product interaction at the shelves
For more detailed information, shopper behavior in front of shelves can also be observed. The options range from relatively coarse distinctions between passing vs. stopping in front of a certain shelf to detection of hand movements and interaction with products. Ceiling- or shelf-mounted cameras combined with smart algorithms allow the identification of target behaviors. The results reveal areas that receive exceptional high or low levels of attention in terms of stopping or interaction. For example, they allow to identify products that are often grabbed but then returned to the shelf. Further analysis can reveal why customers abandon the product despite initial interest. Today, judging from our own experience with shelf interaction tracking, such data can still be extremely noisy and needs to be evaluated and analyzed very carefully. However, several tech companies are working on this problem. Amazon GO, for instance, already trusts their shelf-tracking technology enough to use it for computing the total a customer needs to pay.  So, we expect technology to advance relatively soon and allow improved applications for retailers.

Recognizing customers’ faces: Customer profiles without registration
Modern POS systems enable efficient and timely tracking of sales in terms of when and where which products are purchased at what price. What is missing is information about the customer – about who buys a certain product. Even basic customer profiles based on sociodemographic features such as age group, gender, and whether the shopper is alone or in company are helpful for more targeted communication. Already, there are new smart cameras available that automatically analyze recorded faces in terms of likely age and gender. They only keep this meta data while not storing the face itself and comply with the strict EU legislation on data privacy (GDPR), as to which personal data must not be recorded without explicit consent. Equipped and synchronized with, for instance, the certified privacy-by-design solution AVARD of Fraunhofer Institute, POS systems can add customer data to each recorded sales transaction. However, even if such technologies comply with data privacy regulations, they need to be introduced carefully and consider people’s needs for transparency and control.
A German retailer’s implementation of such systems recently caused a public outrage, leading to its abandonment just a few weeks later. To mitigate consumer concerns and resistance, retailers should find and communicate ways to let customers benefit as well.

Automatic detection of emotional states: The emotional side of customer experience
Automatic analysis of camera-recorded faces offers real-time inference of emotion states. In combination with camera-based gaze-tracking identifying the looked-upon products, retailers might, for instance, detect a customer’s need for information and support. In the introduction, the shopper called Paul has likely expressed his confusion about allergens with a puzzled frown while looking at the package. With some decision support at the right time, his choice deferral may have been prevented.A/B experiments in virtual reality for optimizing customer experience and sales. Providers of online shopping websites can easily employ A/B tests to optimize customer experience and turnover. For brick-and-mortar stores, it is much more complicated and costly to experiment with layout, assortment, or prices. Virtual reality (VR) offers retailers environments for controlled experimentation. For example, different shop layouts can be compared relatively fast and efficiently in VR. The big advantage compared to real world A/B tests is the high degree of experimental control over the situation and the opportunity to get extended information about users’ actions and reception of the situation, for example via eye tracking. Our research indicates that VR is well-suited to predict purchase decisions. In cooperation with the Karlsruhe Institute of Technology (KIT) and CITEC at Bielefeld University, we used choice-based conjoint to compare choices made in an online shopping environment on a desktop screen to those made in front of a 3D full-immersive virtual reality supermarket shelf. We evaluated both in terms of accordance with choices made at a real shelf, resulting in comparably high internal and external validity. It should be noted that designing and programming 3D models for VR is still very time-consuming and expensive. However, once programmed, subsequent adjustments in existing virtual worlds can be implemented more easily.

A/B experiments in virtual reality for optimizing customer experience and sales
Providers of online shopping websites can easily employ A/B tests to optimize customer experience and turnover. For brick-and-mortar stores, it is much more complicated and costly to experiment with layout, assortment, or prices. Virtual reality (VR) offers retailers environments for controlled experimentation. For example, different shop layouts can be compared relatively fast and efficiently in VR. The big advantage compared to real world A/B tests is the high degree of experimental control over the situation and the opportunity to get extended information about users’ actions and reception of the situation, for example via eye tracking. Our research indicates that VR is well-suited to predict purchase decisions. In cooperation with the Karlsruhe Institute of Technology (KIT) and CITEC at Bielefeld University, we used choice-based conjoint to compare choices made in an online shopping environment on a desktop screen to those made in front of a 3D full-immersive virtual reality supermarket shelf. We evaluated both in terms of accordance with choices made at a real shelf, resulting in comparably high internal and external validity. It should be noted that designing and programming 3D models for VR is still very time-consuming and expensive. However, once programmed, subsequent adjustments in existing virtual worlds can be implemented more easily.

Augmented reality for convenient and flexible decision support
Augmented reality (AR) means "augmenting" a real-world environment by computer-generated perceptual content or information. AR apps may facilitate in-store navigation and provide efficient decision support by, for instance, highlighting features the shopper cares about. Customers might use their smartphone camera to get additional information on allergens, customer ratings or climate footprint for all products on the shelf. Thus, AR can make information as readily available in stores as in online shopping. Even visual filtering of shelf content is possible with AR. Data about in-store information searches using AR can inform retailers about the shopping mission and the decision process of consumers, and ultimately also make shopping more convenient. KIT researchers are developing AR applications for in-store decision support. One objective is making the system responsive to automatically detected information needs and decision stages of the customer for optimized decision support at the POS.