Using Automated Model Management for CPG Trade Success

CPG executives invest billions of dollars in trade and consumer promotion investments every year, spending as much as 15-20% of their total annual revenues on these initiatives. However, studies show that less than 72% of these promotions don’t break even and 59% of them fail. Despite these troubling statistics, most CPG organizations continue to design and execute essentially the same promotions year after year with negligible hope of obtaining sustained ROI. 

Trade promotions are becoming increasingly complex and harder to manage as CPG organizations struggle to respond to evolving digital channels of engagement, increasing demands from retailers, and ever-changing consumer behaviors. 

Companies are discovering the obsolescence of using manual processes such as basic rules-based technologies and spreadsheets to optimize CPG campaigns. This is due to the proliferation of trade promotions globally, the huge number of SKUs, and constantly evolving channels and consumer preferences. And using these outdated methods to facilitate CPG promotions at scale is virtually impossible. 

The billion-dollar question is: How can CPG players successfully run a profitable trade promotion in the face of today’s complex and diverse CPG industry? 

Thankfully, emerging technology such as AI and machine learning can help CPG companies execute promotions with greater precision to obtain impactful results and higher ROI. CPG organizations are recognizing the value of AI, ML, and other advanced technologies in enhancing product visibility, increasing brand awareness, and overcoming operational challenges. 

To watch the full joint webinar with Tredence about MLOps solutions for the CPG industry, check out the on-demand recording here.

Investing in data-driven TPM solutions that harness the power of AI enables CPG companies to change the way they conduct trade promotions. They can enable the integration of internal and third-party data sources as well as raw unstructured data in real-time to deliver better insight into current trends, consumer insights, and brand loyalty. 

Companies can leverage AI-driven predictive analytics tools to maximize the impact of their trade promotions and drive better outcomes, resulting in higher customer satisfaction and brand engagement. 

Retailers typically use forecast models with 2 -3 years of historical data to predict sales, margins, and volume. However, the evolving nature of the CPG industry means that this data becomes stale pretty quickly, leading to degradation and deterioration in model accuracy, if not maintained. 

Machine learning can continuously tune thousands of models and detect, incorporate and take advantage of new trends/factors. Furthermore, machine learning can connect and inform the three stages of promotion (forecasting, planning, and evaluation) at scale to provide account managers with intelligent insights on the way to execute promotions to maximize ROI. 

Challenges of Optimizing CPG Promotions at Scale 

Despite the above successes, there are still several challenges with maintaining large-scale TPM solutions. One key challenge inhibiting the successful running of trade promotions is the unavailability of first-party data. CPG organizations are forced to rely on data from third-party sources and syndicated providers. Also, ensuring the effectiveness of CPG forecasting models deployed in production requires a mechanism for handling and moving the large streams of data coming in from disparate sources. And doing so at scale is extremely challenging. 

Handling Large, Multi-Model CPG Data 

As such, one of the most critical challenges in streamlining and operationalizing CPG trade promotion solutions is handling and moving data around. A typical TPM data pipeline in production involves the collation of data from a variety of sources…including ETLs from traditional databases or data lakes, data coming from ERP systems, flag files and POS machines, real-time streams from sensor data, etc. While there are several tools for automating and streamlining model training, the real issue is collating, analyzing, and processing the disparate multi-model data collated/generated using different tools and practices across different environments. 

Rapidly Changing Datasets 

Also, like all large-scale ML deployment scenarios, there’s always the danger of concept drift for the forecasting models used in TPM solutions. Business teams need accurate, timely insights to make critical decisions regarding their promotion campaigns. Datasets rapidly change in the CPG industry due to consumer behaviors/preferences, competitor actions, economic pressures, etc. and this reduces the accuracy of the models and by extension, the insights they deliver to CPG organizations. 

Maintaining CPG Models in Production 

Another key challenge of productionizing CPG models is the siloed nature of ML teams. DevOps, data science, and data engineering teams usually work with different sets of tools to process different sources of data from advertisers, merchants, or retailers at different stages in the ML pipeline. 

The question ML teams ask when going into production is, how do we access all those operational datasets that come from different sources in different formats (both online, offline, and real-time data) and bring them all into a single model while monitoring the health of all these models in production to avoid data drift? 

The best way to do this is to take all these disparate data and turn them into features…features that will be used to build data for training, serve the models during production and analyze the results of the served models to detect drift and also explain/predict model behavior over time. 

Doing so would require three different pipelines which would affect model accuracy due to the drift in data across these pipelines. The first pipeline develops the features, the second consumes it and the third brings it to production. It's essential to continuously verify model accuracy to prevent drift and ensure ROI through constant tracking and monitoring of models to ensure that training and serving data is identical…which can be quite difficult in a distributed environment. 

Solving these challenges requires a seamless way to harmonize and handle data uniformly irrespective of source, APIs, tools, or methodologies in use…and this is where the concept of a feature store comes in. Data scientists need to consolidate the online/offline feature processing and feature serving infrastructure and then go beyond this to create a federated feature store to handle all the data that’s coming from various environments. 

As such, CPG organizations need an efficiently holistic data science platform — one that simplifies the data management process, performs training, inference, validation, and refinement of ML models, and lowers the cost of collecting, ingesting, classifying, and preparing data. 

MLOps in CPG Promotions 

Instant analysis of streams of data obtained from various sources and data pipelines can arm CPGs organizations with prescriptive recommendations on where to spend trade dollars more effectively. As such, adopting MLOps is a great way to handle and move the large and rapidly evolving datasets required by forecasting models to ensure the accuracy and timeliness of trade promotion insights. It can be used to build and manage predictive models that inform every stage of CPG trade promotions: forecasting, planning, execution, and evaluation. 

MLOps can help: 

• Deliver accurate insights in real-time to help CPG companies understand the big picture 
• Provide 360-degree visibility into spend and competitor activity. 
• Ensure accurate post-event analysis for improving promotion performance. 

CPG organizations can leverage an MLOps solution to manage their forecasting models right from building and testing all through to deployment and monitoring. The solution should come with an operationalized pipeline that includes a drift detection mechanism that kicks in immediately during production to analyze and inform teams when there are differences in the distribution of data from that point compared to when the model was trained. 

Essentially, such a solution would bring together all that data, and then crunch, aggregate and optimize it for training and then serve the data to the thousands of models required to forecast in AI-driven TPM solutions as well as handle all drift detection and governance issues 

With the Iguazio Data Science Platform, you have access to a fully automated pipeline that can take the features from the feature store, build the model for you and create the model serving functions. CPG organizations can use Iguazio to solve the complexities of machine learning operations (MLOps), and accelerate the machine learning workflow by automating the following, end-to-end processes: 

• Data collection: ingested from any diverse source, whether structured, unstructured, raw, or real-time 
• Data preparation: through exploration and manipulation at scale (go big!) 
• Continuous model training: through acceleration and automation 
• Monitoring and management of AI applications in production 

Wrapping Up 

Ensuring successful trade promotion campaigns requires CPG organizations to combine AI-powered tools with human insights to drive accuracy and efficiency for their trade promotion campaigns. These tools provide CPG players with superior data analysis that will enable them to execute trade promotions with far greater precision and maximize the efficiency and ROI of trade promotion spend. 

Iguazio’s Data Science Platform streamlines the entire MLOps process to help CPM solutions achieve automated, full lifecycle management of CPG forecasting models. This enables CPG organizations to explore the ramifications of thousands of alternative scenarios, explore trade-offs across variables (channels, retailers, SKUs, etc.), and consider millions of possible combinations to deliver optimal recommendations for trade promotion.  

To learn more about the Iguazio Data Science Platform, or to find out how we can help you bring your data science to life, contact our experts.