UpTrain Statistics 👨‍🔬

Calculate standard (and custom) metrics over your data

Distances

UpTrain provides various measures to analyze the distance of embeddings. These measures can help in determining whether the model is overfitting or underfitting, and how much the embeddings are changing over time. UpTrain currently supports the following measures for measuring distances between data points:

1. Norm ratio: This measure calculates the ratio of the current embeddings' norm to the initial embeddings' norm. If the norm ratio is close to 1, it indicates that the embeddings have not changed much from the initial embeddings. A large change in the norm ratio can indicate that the model is overfitting or underfitting.

2. L2-norm distance: This measure calculates the L2-norm distance between the current and initial embeddings.

3. Cosine distance: This measure calculates the cosine distance between the current and initial embeddings. Unlike the norm ratio and L2-norm distance, cosine distance considers the direction of the embeddings.

The following is how we can define the check for the distance between embeddings

distance_check = {
    'type': uptrain.Statistic.DISTANCE,
    "model_args": [{
        'type': uptrain.MeasurableType.INPUT_FEATURE,
        'feature_name': 'model_type',
        'allowed_values': ['batch', 'realtime'],
        }],
    'reference': "initial",
    "distance_types": ["cosine_distance", "norm_ratio", "l2_distance"],
}

Here, model_args defines the models that we want to compare, and the reference is the initial embedding (another option is the running difference). Further, we calculate all three distance_types defined above.

In the following figure, we check the comparison between the two model_typesin terms of cosine distance from the initial embedding, and note that for the realtime model, the learning happens much earlier compared to the batch model.

Cosine distance of embeddings from initial embeddings as they are being learnt with time.

In addition to the above measures, UpTrain also provides the running difference of the embeddings. This is the difference between the current embeddings and the previous embeddings. By analyzing the running difference, we can determine how much the embeddings change over time. A large change in the running difference can indicate that the model is experiencing significant changes in the input data or that the optimization process is unstable.

Overall, these measures can help in analyzing the stability and performance of the model's embeddings over time. By monitoring these measures, we can detect issues such as overfitting, underfitting, or instability, and take corrective actions to improve the model's performance.

Convergence analysis

UpTrain also provides convergence analysis for embeddings, a technique for evaluating the performance of an embedding algorithm, and involves measuring how well the embeddings converge as the algorithm iterates. UpTrain provides several methods for conducting convergence analysis on embeddings, including visualization tools and metrics that can be used to evaluate the quality of the embeddings.

The following is how we can define the config to check for convergence statistics:

convergence_check = {
    'type': uptrain.Statistic.CONVERGENCE_STATS,
    "model_args": [{
        'type': uptrain.MeasurableType.INPUT_FEATURE,
        'feature_name': 'model_type',
        'allowed_values': ['batch', 'realtime'],
        }],
    'reference': "initial",
    "distance_types": ["cosine_distance", "norm_ratio", "l2_distance"],
}

In our dashboard, we observe that at time 100k, the norm ratio for embeddings generated by the batch model is higher, implying that there is a greater popularity bias.

Norm ratio of embeddings w.r.t. initial embeddings at time 100k.