Irrelevance detection in NLU

  • Release version: Yokohama
  • Updated January 30, 2025
  • 5 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Irrelevance detection in NLU

    Irrelevance detection in NLU helps improve Virtual Agent chat performance by training the NLU model to ignore user utterances that do not correspond to any defined intent. This ensures the model does not make incorrect intent predictions for irrelevant inputs, keeping conversations focused and accurate.

    Show full answer Show less

    Key Features

    • Marking Utterances as Not Relevant: Utterances that should not trigger any intent can be labeled as Not relevant and included in model training to avoid predictions on such inputs.
    • Access and Roles: Available only for Virtual Agent models, accessible via the nluadmin, admin, or nlueditor roles with appropriate model assignments.
    • Adding Irrelevant Utterances:
      • From Virtual Agent chat logs via the Expert Feedback Loop.
      • Manually by typing directly into the Irrelevance detection interface.
      • By importing CSV or Excel files with empty Intent columns.
    • Management and Limits: Models can have up to 200 irrelevant utterances combined from those specific to the model and those irrelevant to all models. If exceeded, the oldest irrelevant utterances are dropped.
    • Conflict Handling: Irrelevant utterances override conflicting intent utterances, preventing predictions. Conflicts are visible in the Cross-model Conflict Review module and intent Conflicts tab, where irrelevant utterances appear as NOINTENT.
    • Reassignment and Editing: Irrelevant utterances can be reassigned to intents if needed, and must be saved and followed by retraining to update the model. They can also be edited or deleted from the Irrelevance detection table, but not from the conflict pages directly.

    Practical Guidance for Customers

    • Include about 10% irrelevant utterances in test sets to evaluate how the model handles out-of-scope inputs.
    • Use the Irrelevance detection feature to maintain high prediction accuracy by preventing false intent matches on irrelevant user inputs.
    • Regularly monitor conflicts involving irrelevant utterances to ensure your intents and irrelevant utterances remain distinct and do not overlap.
    • Retrain your Virtual Agent NLU models after adding or updating irrelevant utterances to incorporate these changes.

    Key Outcomes

    By effectively utilizing irrelevance detection, ServiceNow customers can maintain more focused and accurate Virtual Agent conversations, reduce false intent matches, and improve overall user experience by ensuring the NLU model responds appropriately only to relevant utterances.

    Keep Virtual Agent chats focused with Irrelevance detection. Use the Irrelevance detection feature to train your NLU model to avoid making predictions for utterances that are not relevant.

    Summary information

    The Irrelevance detection feature improves the prediction accuracy of NLU models by training them to ignore certain utterances. These utterances from your users may not apply to any intent, so should not get any prediction.

    To ensure that models do not predict an intent when they are not supposed to, you can mark utterances as Not relevant. These marked utterances are included as part of model training. When the published model encounters similar utterances from your users, no intent is matched or predicted.

    The Irrelevance detection table is where you can manage utterances marked as Not relevant.

    Roles, Usage, and Navigation

    Use the nlu_admin or admin role to access Irrelevance Detection. The nlu_editor role is also able to access Irrelevance Detection, but must be assigned to a model to edit that model's contents.

    Irrelevance Detection is available for Virtual Agent models only.

    1. Navigate to All > NLU Workbench > Models. The Virtual Agent tab opens by default.
    2. Scroll down the list of Virtual Agent models to the Boost your model performance section.
    3. Scroll horizontally to locate the card Keep chats focused, and select its button Go to irrelevance detection.In NLU Workbench, on the Virtual Agent tab, the card Keep chats focused is highlighted.
    Note:
    The instance URL for this feature is <instance-name>.servicenow.com/now/nlu-workbench/irrelevant-utterances.

    Adding utterances to Irrelevance detection

    There are several methods for adding utterances to Irrelevance detection:

    • The Virtual Agent chat log: In the Expert Feedback Loop in NLU Workbench, when you review an utterance collected from the Virtual Agent chat log, you can mark it as Not relevant. The system will ask whether it should be irrelevant to a particular model, or to all models. After adding, these utterances may display as NO_INTENT in Expert Feedback Loop.

      For more information on Expert Feedback Loop in NLU, see NLU Expert Feedback Loop.

      These utterances have a Source of VA Chat Logs in the Irrelevance detection table.

    • Manual input: In Irrelevance detection, type your utterance in the Type utterances here field, then select Add. The system will ask whether it should be irrelevant to a particular model, or to all models.

      These utterances have a Source of Manual in the Irrelevance detection table.

    • Importing: When you use a CSV or XLSX (Excel Workbook) file to import training utterances and their intents, you can indicate irrelevant utterances by leaving the Intent column empty. These utterances may display as NO_INTENT in areas such as Expert Feedback Loop and Conflict Review.

      For more information on importing utterances and intents, see Create an NLU model from a CSV file.

      These utterances have a Source of Manual in the Irrelevance detection table.

    Behavior of irrelevant utterances

    There are two types of irrelevant utterances: those associated to one specific model, or those irrelevant to any model. A model can have a maximum of 200 irrelevant utterances associated to it.

    Following are details of how these two types and the 200-count maximum interact.

    When a model is submitted for training, at most 200 irrelevant utterances are submitted with it. First, irrelevant utterances directly associated to that model are submitted. Then, utterances which are designated as not relevant to any model are submitted. The total of these types does not exceed 200.

    If a model has 200 irrelevant utterances associated to it, and a new irrelevant utterance is added, then the model's oldest utterance is dropped. The new irrelevant utterance could be of either type (not relevant to the specific model, or not relevant to any model).

    A model cannot have more irrelevant utterances than normal training utterances.

    Conflict review

    If an utterance is marked as irrelevant, and there is a similar utterance in an intent, the model does not make a prediction for that utterance. In other words, irrelevant utterances take precedence over training utterances.

    Because irrelevant utterances impact the model's predictions, they are displayed as conflicts when they overlap with training utterances.
    There are two locations where conflicts with irrelevant utterances are highlighted:
    • The Cross-model Conflict Review module. For more information about this module, see Cross-model Conflict Review.
      Note:
      The Cross-model Conflict Review module is available with the NLU Workbench - Advanced Features application from ServiceNow® Store. For more information, see NLU Workbench - Advanced Features.
    • The Conflicts tab of an intent. For more information on conflicts in an intent, see Resolve intent issues.
    For the purpose of reviewing conflicts, irrelevant utterances are displayed as though they are in their own intent, named NO_INTENT.

    Note that irrelevant utterances cannot be edited or deleted on the conflict page. Copy the irrelevant utterance from the conflict page to the Irrelevance detection page to modify or delete the utterance.

    More information

    • In test sets, aim to include about 10% of test utterances as irrelevant. This helps to assess how your model handles utterances that should not have an intent predicted. When you import training or test utterances from a CSV or XLSX (Excel Workbook) file, you can indicate that an utterance is irrelevant by leaving the Intent column empty.
    • When testing models against test sets, results are considered Correct if no intent is predicted for an irrelevant utterance.
    • Utterances that were marked as Not relevant can be re-assigned later. For example, if a new intent is created, existing irrelevant utterances can be manually assigned to the new intent. They then become part of the normal training utterances.

      To re-assign an utterance in Irrelevance detection, expand the list in the Corrected intent column and select the appropriate intent. Be sure to select the Save feedback button after re-assignment. Also be sure to retrain the model to incorporate these updates into the model.

    • Irrelevant utterances are not associated to particular intents within a model. They are either associated to one model, or marked as not relevant to any model.
    • The utterances which are marked as not relevant to any model are submitted as part of the training data for each model. In other words, these apply to all models.
    • Model training is necessary to incorporate Not relevantutterances. Training any model adds newly marked utterances to all models.
    • Utterances can be deleted or edited in the Irrelevance detection table.
    • Irrelevant utterances should have content that is different from utterances associated to an intent.