Shift the Focus: From Asset to Model Management
By Carolyn Saplicki, Senior Data Scientist, and Erika Agostinelli, Senior Data Scientist
As industries begin to utilize AI in Sustainability practices, the focus, thus far, has been on Asset Performance Management (APM). In this process, statistical or machine learning (ML) models have been created to aid with predictive maintenance.
This means predictive maintenance depends on the health of your models. Therefore, to ensure the best performance of your assets, AI/ML models must have integrity and resiliency. In this blog, we will discuss how can we ensure our APM models are consistently functioning properly and who needs to ensure the credibility of our models. Additionally, we will highlight the importance of having a model management view in industry.
The Advancement of AI in Industry
AI can augment asset performance practices to increase utilization of assets and materials. In APM, AI models could include failure prediction, anomaly detection, end-of-life prediction and many others. Often, one asset may have multiple models. The benefits of using a model to aid with APM include transparency into operations, reduction in asset replacement cost and optimization of maintenance planning.
We would like to draw a comparison between the asset focus approach (manufacturing process) and model focus approach (ML lifecycle management process). Just as a manufacturing process is installed to ensure the quality and consistency of every manufactured good, a ML lifecycle management process must be appointed to ensure good quality and consistency of every deployed model.
We dissect each approach to find common elements such as process inputs, controlled variables, uncontrolled variables, and process outputs. This will highlight similarities, differences and the complementary nature of the two approaches.
Table of contents:
Asset Focus — Manufacturing Control Process
- Benefits of monitoring the Manufacturing Control Process
- Example on Maximo Application Suite
Model Focus — ML Lifecycle Management Process
- Benefits of monitoring the ML Lifecycle Process
- Example on Watson OpenScale
Conclusions
Asset Focus — Manufacturing Control Process
Problems in industry revolve around the manufacturing process. This process consists of using raw materials to create and output finished goods. For instance, in the automotive industry, raw materials such as steel and rubber and energy from fossil fuels are used to create a finished product, like a car tire. Below is an overview of the manufacturing process.
Process Inputs:
- Materials: raw materials needed to create the manufacturing product
- Energy Sources: raw energy sources used to power the manufacturing process
- Human Intervention: human intervention needed to run the manufacturing process
Controlled variables:
- Asset properties: asset properties that determine the functioning of the asset that are individually controlled for the specific process (machine settings)
Uncontrolled variables:
- Environment: variables which we do not have any control, or would have limited control over, and may affect the manufacturing process production
Process Outputs:
- Final good: final good that arises from the manufacturing process.
- Other output: I.E., Yield (proportion of good items), Waste (scrap items), Downtime (the period during which a machine or equipment is not functional)
Benefits of monitoring the Manufacturing Control Process
With a manufacturing control process, operations managers and engineers can take immediate action on:
- reducing downtime and cost, making the APM more efficient
- extension of the asset lifecycles through financial and performance analytics
- optimization of maintenance work processes
Together operations managers and engineers can support asset work orders using insights from anomaly detection, time to failure, and failure prediction models. Finally, this process can manage maintenance with work management for planned and unplanned activities.
Example
IBM Maximo® Application Suite (MAS) is a set of applications for asset monitoring, management, predictive maintenance, and reliability planning. Using IBM Maximo Predict, we use AI and process inputs (raw materials), controlled variables (IoT sensors data), uncontrolled variables (environmental data) and process outputs (final products) to predict downtime, degradation, and failures and to track imminent failures and maintenance schedules for oil wells.
Based on our monitors, we can investigate assets and their respective predictive maintenance models. Through the manufacturing control process, operations engineers investigated:
a. current probability of different failure modes,
b. the number of days until a specific failure mode might occur, and
c. if detected anomalies signal a probable failure.
Specifically, Maximo Predict highlighted an anomaly threshold being violated. This could signal a probable failure. A work order was created, and an engineer investigated the asset to understand the anomaly. This monitor helped uncover a needed maintenance for this asset.
Model Focus — Machine Learning Lifecycle Management Process
The ML lifecycle management process can be compared to the manufacturing process. Just as a manufacturing process has inputs, variables and outputs, a model mirrors this dynamic. The model focus approach consists of using raw data as input to create predictions. For instance, in APM within the automotive industry, raw IoT sensors data is used to predict assembly line failures. Below is an overview of the machine learning process.
Process Inputs:
- Payload Data: live sensor data to be scored utilizing the machine learning model (predictive model such as failure prediction, anomaly detection or end-of-life models).
Controlled variables:
- Internal: variables internal to the model that are required for predictions such as hyperparameters. These can be estimated or learned from the data. These are usually created by Data Scientists.
- External: variables external to the model that are required for model understanding such as usage or monitor thresholds. These are usually determined by Subject Matter Experts (SMEs) and Data Scientists.
Uncontrolled variables:
- Causal: variables part of the asset which is not functioning correctly (faulty sensor of the asset is recording unreliable data which may have an unwanted impact on the performance of a specific asset(s)).
- Non-Causal: variables not part of the asset which we do not have any control (limited control) over. For example, unexpected characteristics of the building (humidity) which is not captured by the model may have an unwanted impact on the performance of assets in different locations.
Process Outputs:
- Predictions: final output of a model (anomaly score, time to failure prediction etc).
- Monitors: outputs from model monitors such as quality, drift, fairness and explainability providing insights into the overall model and final prediction.
Benefits of monitoring the Machine Learning Process
With a machine learning lifecycle management process, machine learning engineers and data scientist can take immediate action on:
- ensuring the quality and performance of the predictive models for an efficient APM
- maintaining resiliency of model predictions through root cause analysis and explainability
- capturing if data drift or model drift is occurring
Together, data scientists and operations can increase accuracy of predictions by identifying how AI is used and where it is lagging, enhancing collaboration across different teams. Finally, this process can mitigate potential performance issues with root cause analysis and model prediction explanations, supporting asset work orders.
Example
In our previous blog, AI Fairness in Industry, we demonstrated the use of a fairness monitor in an industry related problem. We utilized model monitoring on our Prediction Maintenance Model by choosing a machine learning lifecycle management view. Specifically, we wanted to investigate if certain controlled variables (fairness thresholds) were being violated based on uncontrolled non-causal factors (regional humidity). We were able to quantify this bias using our process inputs (payload data; region) and process outputs (predictions and monitors).
To do this, we created a custom metric to uncover possible bias within our model. Using a fairness ratio, we were able to highlight whether the model is biased toward one region in certain asset conditions. In fact, based on historic data, the model learned that one region’s well was going to produce less faulty products than the other region on equal terms (i.e. despite having the same control variables). Below, you can see the fairness metric in Watson OpenScale over time.
Based on our monitor, we can initially see the metric is violating the lower threshold consistently until 12 PM. Through the machine learning process, Data Scientists, SMEs and Operations Engineers investigated the violations and saw if they could be explained by the model and/or mitigated through Machine Learning practices. After consistent violation, the source of the bias was found. The humidity of the building was causing unexpected results from our models. This monitor helped uncover a needed strategy change for this asset class.
Conclusion
It may seem like a machine learning lifecycle management process view with respect to your assets is not relevant at the industry level. However, model understanding is critical in the industry. We are living in an era where we rely on AI to make our assets more durable. It is important to care for those models as much as the assets and apply an approach which focuses on the ML lifecycle management.
The title of this blog “Shift the Focus: From Asset to Model Management” does not mean that assets (machines) are not at the heart of the APM. Instead, it highlights that a complementary strategy is needed to support the robustness of our AI/ML models. The combination of these two approaches will lead to more efficient APM decisions, revealing when strategy changes are necessary.
We recommend businesses to rethink their asset performance management strategy and include ML lifecycle management to optimize utilization of their assets and models.
