NLU model settings

  • Release version: Xanadu
  • Updated August 1, 2024
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of NLU Model Settings

    ServiceNow customers can configure key settings for their Natural Language Understanding (NLU) models via the NLU Workbench. These settings include modifying the model's name, description, business area, and confidence threshold. However, language, purpose, and scope are fixed and require creating a new model if changes are needed.

    Show full answer Show less

    Model Configuration

    • Accessing Settings: Navigate to All > NLU Workbench > Models, select your model’s application, then the model name, and open the Model settings tab.
    • Editable Attributes: Change the model’s name, short description, and business area to align with your organizational needs.
    • Ignore Punctuation: Enabled by default to reduce variance in intent predictions caused by punctuation differences, improving model consistency. It is recommended to keep this enabled for best results.

    Confidence Threshold Settings

    The confidence threshold controls when an intent is predicted based on the confidence score for an utterance:

    • Threshold Definition: A percentage value that must be met or exceeded for an intent to be predicted.
    • Impact of Threshold:
      • A low threshold may increase false positives (incorrect intent matches).
      • A high threshold may exclude valid intents.
    • Setting Types:
      • Automatic: The system dynamically selects the optimal threshold during model testing and publishing, adjusting based on test results.
      • Manual: Users can manually define the threshold and choose to accept system recommendations made during testing.
    • Prebuilt Models: Come with a pre-tuned threshold optimized for their specific use case.

    Threshold Recommendations and Testing

    The system provides threshold recommendations when the following conditions are met:

    • The test coverage is at least 60%, with a minimum of 5 utterances per intent.
    • The test set contains at least 100 utterances.
    • The model is custom-built (not prebuilt).
    • The recommended threshold improves prediction accuracy compared to the current setting.

    When recommendations are available, a second graphic displays predicted intent percentages with the recommended threshold. Customers can apply these recommendations to automatically retrain the model and enhance prediction performance.

    Change your NLU model's name, description, or confidence threshold on the Settings page of the model overview.

    Access the model's settings by navigating to All > NLU Workbench > Models. Select the tab for your model's application, then your model's name. On the model's overview, select the Model settings tab. Model settings on the model's overview page

    Model settings

    In the upper section of the model settings page, you can change the model's name, short description, and business area. You cannot change the model's language, purpose, or scope. To make a model with a different language, purpose, or scope, see Creating models.

    By default, the Ignore punctuation check box is active. Ignoring punctuation makes it so that there is less variance between predicted intents and confidence scores for utterances with slightly different punctuation. For best results, keep the check box active.

    Model threshold settings

    Here you can adjust how the confidence threshold works in your model.

    A threshold is a confidence score represented by a percentage. The confidence threshold of a model determines what intents from that model will be predicted for a given utterance. For example, if the model threshold is 65%, then an intent will be predicted for an utterance only when the intent has a confidence score that is at least 65%. Setting a threshold that is too low may increase the false positives by predicting intents that should not be a match for an utterance. On the other hand, a model threshold that is too high may filter out intents that you do want to get predicted. Finding the ideal threshold improves your model's ability to predict intents correctly.

    There are two types of model threshold settings:
    • Automatic - Allow the system to choose the optimal confidence threshold for your model. The value is updated dynamically based on test results. This happens in the Test and publish your model phase, where your model's default test set is used.
    • Manual - You can manually set the confidence threshold. The system may also recommend a better threshold for the model during testing. You can choose to accept recommendations.

    Prebuilt models come with a tuned threshold. The confidence threshold on prebuilt models was chosen specifically for that model.

    Test results include a model threshold recommendation only if they meet the following requirements:
    • The test set has a Test Coverage score of at least 60%, with at least 5 test utterances per intent. For more information, see Test set creation and management.
    • The test set has at least 100 utterances.
    • The model is not a prebuilt model.
    • The recommended threshold would have better results than the current threshold.
    Test results page with the bar charts for the current and recommended thresholds.

    Test results with a recommended threshold contain a second graphic. The second graphic shows the prediction percentages with the recommended thresholds applied.

    Applying the threshold recommendation may improve the prediction percentages of your model. Select Apply recommendations to change the threshold. The system automatically retrains the model, and the test results show the prediction percentages with the new threshold.