The objective is to be able to rapidly, reliably and repeatedly deliver value in production, and the unification of dev and ops has become a technological revolution commonly referred to as (perhaps unsurprisingly) DevOps.
A more recent trend in industry has been MLOps - extending the philosophy of DevOps to Machine Learning, typically through unifying data scientists and operations teams, to achieve similar goals. Because of the ways Machine Learning works, however, there are a few additional complexities and nuances that need to be considered when planning a move to MLOps:
The Machine Learning process is inherently experiment-oriented - fundamentally we are attempting to prove or disprove whether ML can solve a business problem. This might mean, for example, by beating the performance of the existing business process, or through an R&D-heavy undertaking to understand whether ML can apply to a problem at all or not. Although agile software deployments are more test-driven than more traditional models, the intended outcomes of DevOps and MLOps are still different.
Where normal, deterministic software can be easily regression tested, it is inherent in machine learning models that we want outputs to change with improving performance. As an example, if a previous model predicted customer churn with 85% accuracy, and produced a score for a user of 0.783, we might have a new algorithm that predicts with improved accuracy, and for the same user produce a score of 0.832 - regression testing against a moving target is inevitably challenging.
This is especially important to consider in models that deal with human behaviour. Data and models inevitably drift - for example with customers altering their behaviours because of working from home, or because of changing trends in retail. It’s more critical than ever to be able to deploy new models at speed.
Monitoring loops start beyond software monitoring and logging - depending on the problem, feedback loops can range from seconds to months (for example, in applications such as fraud detection). The choice of metrics will depend heavily on the context of the business problem.
While MLOps does adapt two concepts from DevOps - Continuous Integration (CI) and Continuous Delivery (CD) - it also then needs a further pillar to account for the differences between development and machine learning. This is known as Continuous Testing (CT) and allows for both testing and retraining of models, giving us the CI/CD/CT approach. By appreciating the subtle differences and by adopting CI/CD/CT, teams can realise significant improvements with MLOps.
Whilst innovation in this space is far from finished, and will continue to see the creation of new tools and ideas for many years to come, MLOps has been able to introduce some particularly useful real-world concepts and technological developments in recent years.
A substantial amount of effort has also gone into model explainability, which we will explore in subsequent blogs. Thanks for reading!
The Analytics Revolution blog series: