CIO
Domino's orders up MLOps to speed up data science delivery
By: Clint Boulton
As much as any big brand, Domino’s Pizza leans heavily on data to improve customer service. But the data science team at the world’s No. 1 pizza maker employs a secret ingredient to augment its decision-making: a machine learning (ML) hosting platform that helps it deliver models to production more quickly.
The team uses the platform to run, refine, and validate multiple ML and artificial intelligence models while reducing its reliance on IT for compute resources, which had historically been a critical friction point, according to Zack Fragoso, Domino’s manager of data science and AI.
“People like pizza and we process a lot of orders,” says Fragoso. “The missing piece to the puzzle was a way to get models into production.”
Harnessing analytics for business insights is a hard enough discipline, but data science practitioners must also contend with delays procuring everything from servers to software development environments from IT. Such delays pose problems for data science teams building potentially time-sensitive data models to augment their decision-making and achieve the desired business outcome, Fragoso says.
Domino’s journey to MLOps
Domino’s, which totals $14 billion in sales a year, has mitigated this challenge in part by relying on five-year-old startup Datatron, which competes with DataRobot, Google, and others in building platforms that help companies operationalize ML models, also known as MLOps.
Datatron is a fairly new addition to Domino’s data science recipe. When Fragoso joined the data science team as employee No. 7 in 2018, Domino’s executed models for its ecommerce system the traditional way. The team would wait for daily data extracts from Domino’s stores to retrain a data model and then file a ticket with IT for the resources to run the refreshed models with the most up-to-date data. But even that relatively short 24- to 48-hour lag from model to production impeded data accuracy.
“We were handcuffed,” Fragoso recalls. “There were a lot more touchpoints and time doing updates that didn’t need to be there.”
Today, the data science team boasts 63 members, who deploy, run, and recalibrate ML models with Datatron and Kubernetes container orchestration software running in Domino’s private cloud. The software’s visualization capabilities help the team understand how predictions for customer experience and store operations change over time and enables them to refresh those models based on new data, Fragoso says.
For instance, Datatron enables the team to run A/B tests in which it routes a small amount of traffic to the “B” challenger website to validate the new models — all without impacting the ecommerce team or store operations stakeholders. “We’re continuously improving, developing, and updating models as new data comes in,” Fragoso says.
Ultimately, the data science team’s work with Datatron has helped to improve labor scheduling by forecasting staffing requirements for each store, to optimize vehicle routing for delivery drivers, and to sniff out locations to build new stores, among other targets. It has also helped Domino’s track the data models for drift, a common occurrence in store operations, which tends to be dynamic. “If any piece changes, the models need to be updated to reflect the new reality,” Fragoso says.
The implementation of most new complex technologies can be fraught with tension, requiring a massaging of egos that is typical of change management. And so it was with Datatron at Domino’s, where IT was traditionally heavily involved in the data science team’s production efforts. But solid results helped the data science team cut through the “healthy apprehension,” Fragoso says.
“It’s a win-win for data science and IT,” Fragoso says. “The data science team is much more engaged, while IT is not forced to dedicate resources to field requests from data science.”
The ML operations struggle is real
Domino’s data science team has plenty of company in the struggle to build data models and execute them in the real world. Fifty percent of IT leaders will struggle to move their predictive projects from proof of concept to production by 2023 thanks to technology hurdles, poor data quality, and other snafus, according to Gartner research published in 2020. Interdepartmental politics also muddies matters, requiring strong collaboration between data scientists, source system experts, system architects, application developers, and other specialists.
While many data scientists hail from a variety of backgrounds, including statistics, physics, and other disciplines, most don’t have computer science or programming experience, let alone the capability to work with software infrastructure and processes that would enable them to scale their models, says Datatron CEO Harish Doddi.
Conversely, most infrastructure engineers lack experience building data models, which is why collaboration and negotiation is required between the data science and IT, says Doddi, who saw such knowledge gaps while working in software engineering roles at Lyft, Snap, and Twitter before founding Datatron.
In the typical AI model lifecycle, data scientists develop models and hand them off to infrastructure engineers, who launch them into production. Datatron’s software automates much of the technology blocking and tackling that data science teams perform to run their SAS, R, ML and regression models. It also helps monitor the models in production, governing against “model drift,” in which the predictive properties ebb as data changes.
“The AI models are only productive if they make it into production,” says Doddi.
MLOps can help data scientists construct that bridge.
