Configuring target metrics for a trained classification solution

  • Release version: Xanadu
  • Updated August 1, 2024
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Configuring Target Metrics for a Trained Classification Solution

    This guide explains how ServiceNow customers can configure target metrics—precision, coverage, and recall—for trained machine learning classification solutions using the Solution Statistics tab. Adjusting these metrics at the solution level helps optimize model performance based on business priorities, while the system automatically updates class-level metrics accordingly.

    Show full answer Show less

    Configuring Solution-Level Metrics

    • Metrics Explained: Precision reflects prediction accuracy, coverage indicates the proportion of data predicted, and recall measures identification completeness.
    • Setting one metric impacts the others; changes are applied iteratively and recalculated in real time.
    • Users select a target metric and specify a percentile value between 0-100, then apply it. The system trains predictions to favor the selected metric while balancing trade-offs among the others.
    • Automatic propagation of solution-level metric values to individual class-level statistics simplifies management, especially with many classes.

    Practical Configuration Steps

    1. Navigate to the Solution Statistics tab of a trained classification solution.
    2. Review the informational banners explaining each metric to understand target value implications.
    3. Select a target metric from the choice list and enter a desired percentile value.
    4. Click "Apply Values" to update the solution’s estimated precision, recall, and coverage metrics.
    5. Review how the system recalculates and balances these metrics based on your input.

    Use Case Examples

    • Precision Focus: For automating incident routing with a goal of ≥80% correct predictions, setting precision to 80 resulted in a slight exceedance (80.04), meeting business needs.
    • Coverage Focus: Targeting at least 70% incident prediction coverage increased coverage from about 36% to 56%, but reduced precision, illustrating trade-offs influenced by data distribution.
    • Recall Focus: In security scenarios like phishing detection, setting recall to 95 improved recall but lowered precision, reflecting the priority to minimize false negatives.

    Class-Level Insights

    After applying solution-level metric targets, customers can view the resulting class-level precision, coverage, and recall statistics. Sorting classes by precision helps identify which classes perform best under the current configuration. Iterative adjustments allow fine-tuning until the desired balance across metrics and classes is achieved.

    Why It Matters

    Configuring target metrics enables ServiceNow customers to tailor machine learning classification solutions to their specific operational goals, such as prioritizing accuracy, coverage, or recall depending on use case requirements. This flexibility ensures the solution aligns with business priorities and maximizes value from predictive intelligence models.

    Set values for precision, coverage, and recall statistics for a trained machine learning solution.

    Setting classification metric values at the class or solution level

    Predictive Intelligence provides three classification metric types: precision, coverage, and recall. You configure these metrics on the Solution Statistics tab of a trained classification solution form. While you can manually set values to these metrics at the class level, doing so can be challenging if you have a large number of classes to cover. In many cases, you may not know the best value to set until your solution is trained. This topic focuses on setting the metric values at just the solution level.

    Configuring solution metrics

    When you apply a value to one metric, it changes the values of the other two. This behavior enables you to modify your metrics iteratively in real time to see which value combinations render particular results. When you apply a new value to a metric, the system recomputes it by considering its new targets.

    Applying a value to a metric asks the system to train its predictions to favor the metric you set based on the highest percentage value, and at a cost to the other metrics. The system tries to meet these values but may not set them exactly as you request due to how the data you're training is distributed.

    When you apply metric values at the solution level, the system automatically sets the appropriate values at the class level.

    Here are the basic steps for configuring a target metric for your solution.
    1. Navigate to the Solution Statistics tab of a trained ML solution.
    2. Review the messages on the green banners of the screen which define each of the metrics so you can better understand the values you want to assign to the solution. The first two message banners address estimated solution-level metrics. The third banner addresses class-level results based on the solution values you applied.
    3. In the Target Metric choice list, select the metric you want to configure.
    4. In the Target Metric Value field, enter a numeric percentile value between 0-100.
    5. Click Apply Values.
    6. Result: On the Solutions Statistics tab, you can review the change in values to the Estimated Solution Precision, Estimated Solution Recall, and Estimated Solution Coverage. The system calculates these values based on the Target Metric you select and the Target Metric Value you enter for the solution.

    Here's a sample landing page for a recently trained classification solution. As you can see, the precision metric is 44.18, recall is 41.26, and coverage is 77.23.

    This sample image shows the estimated values set for solution precision, recall and coverage metrics.

    If you need to adjust these default values for a use case, refer to the sample configurations below. For example, based on the classification solution you're implementing, you might want to change the target metric value for precision, recall, or coverage. Keep in mind that when you change the target metric value for one metric, such as precision, it impacts the values of the recall and coverage metrics as well.

    Precision configuration example

    In this example scenario, you're replacing a manual triage process for routing incident records with an ML classification solution that automatically assigns the records to the correct assignment group. For this scenario, you have a target value in mind and the system must predict correctly at least 80% of the time. So you set the precision metric value to 80 and click Apply Values.

    This image shows you how to set the Precision metric to 80%.

    Here are the metric values the system applied to the solution. In this scenario, the precision value of 80.04 slightly exceeded your request for 80%, so you're likely satisfied with that value.

    This image shows you the estimated precision, recall, and coverage values the system assigned to the solution based on your Precision input value of 80%.

    Coverage configuration example

    In another example scenario where you're replacing a manual triage process for routing incident records, your minimum goal is to predict at least 70% of incoming incidents in the first quarter of the year. So you set the coverage metric value to 70 and click Apply Values.

    How to set the Coverage metric to 70%.

    The metric values the system applied to the solution are shown in the following image. The coverage metric value increased from 35.99 to 55.98. However, the precision metric decreased from 80.18 to 64.97. This could be because you set the coverage metric to a relatively high value of 70, or perhaps because of how the data you're training is distributed.

    The estimated precision, recall, and coverage values the system assigned to the solution based on your Coverage input value of 70%.

    Recall configuration example

    In another scenario, classifying if an incoming email is a Phish or not can be an important use case in a security-related machine learning solution. In this situation, it's very important to identify every Phish, and it may be okay to report a non-Phish as a Phish occasionally. However, no real Phish should be classified as a non-Phish. In such situations, the recall metric must have a high value, which might lead to lower percentages for precision and coverage. So here you can set the recall metric to 95 and click Apply Values.

    This image shows you how to set the Recall metric to 95%.

    Here are the metric values the system applied to the solution. The recall metric value increased from 54.87 to 61.03. However, the precision metric decreased from 60.1 to 55.44. This is likely because you set the recall metric to the high value of 95.

    The estimated precision, recall, and coverage values that the system assigned to the solution based on your Recall input value of 95%.

    Class-level results for the solution metric values you apply to your solution

    The following image shows an example of the class-level results the system applied to a solution's precision, coverage, and recall statistics for 37 classes. You can keep modifying the metric values until you're fully satisfied with the results.

    By Sorting (z to a) on the Estimated Precision column you can see which classes have the highest precision for the solution.

    The class-level results the system applied to a solution's precision, coverage, and recall statistics for 37 classes.