UpTrain Monitors 📊

Monitoring the ML model in Production

Monitors are pre-defined classes in the UpTrain framework that helps observe problems in your machine learning model. UpTrain includes several standard monitoring methods, such as

1. Concept Drift Detection

2. Data Drift Detection

3. Data Integrity Checks

4. Edge Case Detection

5. Quantifying Model Bias

Most importantly, since all machine learning models are different, UpTrain also provides users the ability to define Custom Monitors based on their specific use cases.

The abstract montitor class, called the AbstractMonitor, contains three methods. Thich implies that any custom monitor the user defines should contain these three methods.

class AbstractMonitor:
    def check(self, inputs, outputs, gts=None, extra_args={}):
        raise Exception("Should be defined for each class")

    def is_data_interesting(self, inputs, outputs, gts=None, extra_args={}):
        raise Exception("Should be defined for each class")

    def need_ground_truth(self):
        return False

The first method check() contains the actual check that is to be performed on the model. This check can be data drift (to see if the distribution of production data differs from the distribution of a reference dataset, such as the training data) or concept drift (that is, check if there is a drop in model accuracy). It takes the following inputs:

• inputs : The input data to the model that needs to be monitored.

• outputs : The outputs provided by the model. For many checks, such as data drift on the input data, model outputs are not needed, and this value can be None.

• gts : Ground truth, such as the right class of the image in image classification or whether the user clicked the recommended item. Again, ground truth is not required for all checks (e.g., for measuring data drift on the model inputs).

• extra_args: This contains any extra arguments provided by the user as a dictionary.

The second method, is_data_interesting(), contains checks if the data point is interesting (for example, if it is an edge case encountered by the model, or if it is a part of the dataset that has sufficient drift from the training data distribution).

The third method, need_ground_truth(), tells the framework if this check requires ground truth to execute. Model performance monitors typically require a ground truth, but data drift monitors generally do not.

Next, let's drill down on the predefined monitors in the UpTrain package.

To see custom monitors in action, we recommend checking out the fraud detection example and the shopping cart recommendation example.