Monitor AWS Sagemaker model using IBM Watson OpenScale

Published in

IBM Data Science in Practice

6 min readApr 4, 2023

Introduction

This article shows how to monitor a model deployed on AWS Sagemaker for quality, bias and explainability, using IBM Watson OpenScale on the IBM Cloud Pak for Data platform. You can leverage both AWS Sagemaker and IBM Cloud Pak for Data platforms to bring a machine learning model from development to production with maximum flexibility, and then monitor the health and performance of the deployed models with Watson OpenScale.

This article is the second in a series. The first article, Deploy an IBM Watson Studio Model on AWS Sagemaker, describes how to create a machine learning model using Watson Studio tools on IBM Cloud Pak for Data, then export and deploy the model on Amazon Web Services using Sagemaker. This article shows how to use the endpoint generated from that tutorial to demonstrate how to monitor the AWS deployment with Watson OpenScale.

The sample scenario

The machine learning model created in part one and deployed to AWS Sagemaker predicts how likely a customer is to enroll in a Demand Response Program of a utility company. To build on that, we provide a Jupyter notebook coded to demonstrate how to:

Fetch the demand response model endpoint from AWS Sagemaker.
Configure Watson Openscale on IBM Cloud Pak for Data.
Create a subscription for the AWS Sagemaker model.
Monitor the model for quality, bias and explainability.

You can run the notebook as a hands-on tutorial for setting up the Watson Openscale monitors, or you can review the notebook code and use this article as a guide to the process.

What can you learn from evaluating deployments with OpenScale?

Machine learning models are not a set it and forget it operation. Over time, the performance of a model can change, or poor input data can produce undesirable results, such as bias in predictions or recommendations. Watson OpenScale can be a key part of your AI Governance strategy to provide insights into how the model is performing so you can take steps to update the model when needed. You can configure OpenScale to:

Monitor model predictions to detect bias by comparing outcomes for a reference group to outcomes for a monitored group.
Measure the quality of predictions using a set of standard data science metrics to evaluate how well the model predicts outcomes that match the actual outcomes in the labeled data set.
Review aspects of the model to derive insights. In OpenScale, an explanation helps understand results and also lets you experiment with what-if scenarios to help address issues.

Accessing the sample project and notebook

If you have an IBM Cloud Pak for Data account, you can add or download the sample project from the Gallery to set up and run Watson Openscale to monitor AWS Sagemaker model. The gallery project uses the following platforms:

Watson Studio, Watson Openscale with a Python 3.9/3.10 environment
running on IBM Cloud Pak for Data 4.5.x+, or on IBM Cloud Pak for Data as a Service
Amazon Web Services (AWS) environment with Amazon Sagemaker where a machine learning model is deployed.
DB2 or IBM Cloud Object Storage credentials to store sample training data.

IBM Watson Studio on CPD empowers data scientists, developers, and analysts to build, run and manage AI models and decision optimization solutions in a cloud or on-premises environment. Use Watson Openscale with trust and transparency and understand how your AI models make decisions and get alerts when the performance dips below thresholds you set.

Amazon SageMaker is a machine-learning platform that enables developers to create, train, and deploy machine-learning models in the cloud. SageMaker also enables developers to deploy ML models on embedded systems and edge-devices.

Create a machine model learning model on AWS Sagemaker

Follow the steps in this blogpost to learn how to build a machine learning model on AWS Sagemaker or bring a model from other platforms and deploy it on Sagemaker.

Import the sample project

If you deployed the sample Demand Response Model on Sagemaker use the gallery project Monitor AWS Sagemaker Model on Watson Openscale on IBM Cloud Pak for Data as a Service, or download and import the project zip file into IBM Cloud Pak for Data 4.5.x (or higher) platform.

Monitor Sagemaker model on Watson Openscale using a sample project

If you have imported the sample project you are ready to monitor the AWS Sagemaker model deployment.

From the sample project, open the notebook monitor-sagemaker-model-with-watson-openscale.ipynb in edit mode.

Provide these required inputs to run the notebook:

IBM Cloud API key if you are using Cloud Pak for Data as a Service, or hostname, username and password if you are using IBM Cloud Pak for Data 4.5.x+ platform.
AWS Sagemaker Credentials and the model name deployed on Sagemaker.
DB2 or IBM Cloud Object Storage credentials for the storage repository where you will store sample training data.

2. Run the notebook, using the Run all cells option or run it cell by cell. The notebook performs following tasks:

Fetch the Sagemaker model and deployments using the AWS credentials provided.
Configure the Watson OpenScale service using the IBM Cloud Pak for Data credentials.
Define the Service Provider and Subscription for AWS Sagemaker model on Watson Openscale.
Set up the Quality monitor: Watson Openscale sets an alert threshold for the model and shows an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.

Set up a Fairness monitor: Watson OpenScale helps in detection of Bias at run time. It monitors the data which has been sent to the model as well as the model prediction (Payload data). It then identifies bias in a specified column. In our example we use gender column to check for potential bias in outcomes for the monitored group (female) as compared to outcomes for the reference group (male).

Fairness monitor for GENDER on Watson Openscale

Set up Explanations: Watson OpenScale provides LIME-based and Contrastive explanations for the specified transactions. For example, a sample transaction has a 79.48% confidence score that a particular customer’s response for the demand response program would be YES (1). The Lime analysis chart below shows the top features influencing the model’s prediction. The top features influencing the model’s predicted outcome are HAS_HOME_AUTOMATION, HAS_THERMOSTAT, IS_REGISTERED_FOR_ALERTS, OWNS_HOME and CLTV (Customers Life Time Value).

Explanation for a sample transaction on Watson Openscale

In addition to understanding how your model makes decisions, you can set up what-if scenarios to explore ways you might update the model.

Conclusion:

Combine the power of Watson Studio on Cloud Pak for Data with Sagemaker Studio on Amazon Web Services for maximum flexibility for building machine learning models and bringing them to production. In this blog we reviewed the steps for how to monitor a model deployed on AWS Sagemaker for quality, bias, and explainability, using IBM Watson Openscale. We provided a notebook to configure Watson Open Scale to setup and monitor SageMaker endpoints for Fairness, Quality and Explainability metrics. This example demonstrates how to integrate different services using IBM and AWS to build, deliver, and monitor an end-to-end solution on Trusted AI.

Authors:

Vikram Bhat

Paul Kilroy

Julianne Forgo