NannyML empowers data scientists to detect and understand silent model failure, so you can end these worries in minutes!
NannyML estimates model performance using an algorithm called Confidence-Based Performance Estimation, researched by NannyML core contributors, so you can detect real-world performance drop before you otherwise would.
It can also track the realised performance of your model once targets are available.
NannyML uses Data Reconstruction with PCA to detect multivariate data drift, and for univariate data drift it uses tests that measure the observed drift, and a p-value that shows how likely it would be to get the observed sample if there was no drift.
Model output drift uses the same univariate methodology as for a continuous feature. All of these together help you to identify what is changing in your data and your model.
Target drift is monitored by calculating the mean occurrence of positive events as well as the chi-squared statistic from the 2-sample Chi-Squared test of the target values for each chunk.
ranker = nml.Ranker.by(['alert_count', 'performance_drops', 'feature_importance'])
ranked_features = ranker.rank(drift_results, model_metadata, only_drifting=True)
ranked_features
Because NannyML can estimate performance, it allows you to get alerts on data drift that impact performance. These are tailored to draw your attention to statistically significant events, helping avoid alert fatigue.
You can use our ranker to list changes according to their significance and likely impact, allowing you to prioritise problems. This means you can link drops in performance to data drift that causes it.
NannyML can be set up in seconds on your own local or cloud environments, ensuring your data stays in your control and model monitoring fully complies with your security policies. You can use Pip, Conda, or run it as a CLI.
$ pip install nannyml
Integrates with any classification or regression model, regardless of language or format. More problem types will be supported in future.
Train yourself a model that performs daily predictions of whether your customers will churn in the next three months.
Make sure the model works the way you need it to with the data you have. You can use NannyML here to check the drift in the data you use for validation, and calculate the performance of your trained model on that data.
Get your model into your production environment, collecting new data and making new predictions. You can deploy this anywhere, knowing that NannyML doesn't need access to your model to monitor it, only the data.
Run outputs of your model through NannyML weekly, estimating performance in production. You can also detect any drift on your outputs and features - individually and as a whole.
If NannyML estimates performance problems you can use the tools within NannyML itself to compare different metrics, including drift, and help identify what could be causing the problems. Similarly, if you detect data drift, you can use NannyML to see if the drift is likely to impact the performance.
After you get your targets, you can evaluate NannyML's estimations, and your model's realised performance, by calculating the metrics most important to you.
NannyML turns the machine learning flow into a cycle, empowering data scientists to do meaningful and informed post-deployment data science to monitor and improve models in production through iterative deployments.
Our documentation includes walkthroughs and explanations of how the library works, and how you can use it:
.svg)
If you have any questions about using NannyML, we are happy to help you start monitoring asap!
The team will go through your use cases, and ensure you can cover your basic ML monitoring needs using our library.
Choosing the right tool for you is hard. Here's a starting point for comparing some different ML monitoring solutions.
This is accurate to the best of our knowledge as of September 2022. If you notice anything incorrect, please get in touch!
Check out the open source code in our Github, as well as our detailed readme, code examples and other guides.
Get involved with conversations as part of our community of users, contributors and friends, in our Slack.
Sign up to our newsletter to be kept up to date with future developments of NannyML, and other related data science news we find interesting.
Read our documentation to find extensive tutorials on how to use NannyML, and fully-detailed deep dives into how it all works.
