Detect silent model failure with our Open Source python library

NannyML - OSS Python library for detecting silent ML model failure | Product Hunt

If you like NannyML, join us on Slack!

NannyML Slack community

Performance estimation and monitoring

NannyML estimates model performance using an algorithm called Confidence-Based Performance Estimation, researched by NannyML core contributors, so you can detect real-world performance drop before you otherwise would.

It can also track the realised performance of your model once targets are available.

Data drift detection

NannyML uses Data Reconstruction with PCA to detect multivariate data drift, and for univariate data drift it uses tests that measure the observed drift, and a p-value that shows how likely it would be to get the observed sample if there was no drift.

Model output drift uses the same univariate methodology as for a continuous feature. All of these together help you to identify what is changing in your data and your model.

Target drift is monitored by calculating the mean occurrence of positive events as well as the chi-squared statistic from the 2-sample Chi-Squared test of the target values for each chunk.

ranker = nml.Ranker.by(['alert_count', 'performance_drops', 'feature_importance'])
ranked_features = ranker.rank(drift_results, model_metadata, only_drifting=True)
ranked_features

feature number_of_alerts with_performance_drops feature_importance
0 car_value 2 2 1
1 salary_range 2 1 3
2 driver_tenure 3 0 15
3 loan_length 2 0 9

Intelligent alerting

Because NannyML can estimate performance, it allows you to get alerts on data drift that impact performance. These are tailored to draw your attention to statistically significant events, helping avoid alert fatigue.

You can use our ranker to list changes according to their significance and likely impact, allowing you to prioritise problems. This means you can link drops in performance to data drift that causes it.

Simple and secure

NannyML can be set up in seconds on your own local or cloud environments, ensuring your data stays in your control and model monitoring fully complies with your security policies.

$ pip install nannyml

Integrates with any classification model, regardless of language or format. More problem types will be supported in future.

Screenshot of a jupyter notebook running NannyML

How it fits into your workflow

NannyML turns the machine learning flow into a cycle, empowering data scientists to do meaningful and informed post-deployment data science to monitor and improve models in production through iterative deployments.

A diagram of where NannyML fits into the MLOps process, bridging prediction services and data analysis by providing performance monitoring and post-deployment data science

Based on work by Google

Github

Check out the open source code in our Github, as well as our detailed readme, code examples and other guides.

Slack

Get involved with conversations as part of our community of users, contributors and friends, in our Slack.

Newsletter

Sign up to our newsletter to be kept up to date with future developments of NannyML, and other related data science news we find interesting.

Documentation

Read our documentation to find extensive tutorials on how to use NannyML, and fully-detailed deep dives into how it all works.