Who maintains the maintainers? Implementing predictive maintenance models can significantly increase machinery productivity. However, these machine-learning models tend to fail in time, delivering suboptimal predictions that can lead to many business dollars lost. Monitoring and maintaining them is not trivial because you do not have access to the entire confusion matrix. This happens because the model’s prediction is causing an action to “avoid” maintenance actions. This means that the data scientist has no access to one of real classes (positive or negative, depending on how he/she chooses to encode it), and thus it cannot correctly compute the performance metrics for the class it is “avoiding”.

In this talk, we will focus on using ML monitoring as a critical tool for measuring the quality of the model, identifying root causes, and resolving the issues. We will address three specific challenges of monitoring predictive maintenance models.

In the first part, we will cover how to deal with delayed and partial (AKA censored) target data using performance estimation algorithms that quantify the impact of covariate shift on the ML metrics and allow us to estimate them even without access to target data.

The second part will focus on dealing with low data volume per machine using Bayesian model evaluation and monitoring. This helps to identify model performance issues quickly and is particularly useful when deploying new predictive maintenance models.

The third part will deal with data quality issues and cover simple checks and more advanced covariate drift detection techniques to identify low-quality data quickly.