Legacy - Natural Language Understanding of Virtual Agent responses

  • Release version: Xanadu
  • Updated August 1, 2024
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Legacy - Natural Language Understanding of Virtual Agent responses

    Virtual Agent (VA) leverages the Natural Language Understanding (NLU) service to interpret user input and predict user intents during chat interactions. TheNLU Prediction tabwithin the Conversational Analytics dashboard provides insights into how accurately NLU understands these inputs, helping you to monitor and enhance intent prediction quality.

    Show full answer Show less

    Note that the current Conversational Analytics dashboard is being phased out and will eventually be deprecated. A new Conversational Analytics dashboard is available within the Platform Analytics experience, designed to meet compliance needs such as FedRAMP authorization for Government Community Cloud (GCC). Migration options exist for customers using the legacy dashboard.

    Key Features

    • NLU Prediction Tab: Displays daily performance metrics of NLU predictions with three types of outcomes:
      • Correct Predictions: Multiple intents predicted and user selected one correctly.
      • Incorrect Predictions: Multiple intents predicted but user rejected them all.
      • Auto Selected Predictions: Single intent predicted and automatically selected.
    • Graph Visualization: Shows trends in NLU accuracy over time, helping identify prediction strengths and weaknesses.
    • Data Collection: Tracks whether intents were predicted, matched topics, prediction scores, and user selections to classify prediction outcomes.
    • Model Performance Page: Accessible by clicking the graph, this page summarizes intents predicted or missed during the selected date range.
    • Role Requirement: Access to the NLU Prediction tab requires the chatanalyticsviewer role.
    • NLU Workbench Integration: Enables modification or creation of NLU models to improve prediction accuracy, facilitating ongoing optimization of Virtual Agent interactions.

    Practical Benefits for ServiceNow Customers

    • You can monitor how well Virtual Agent understands user queries and quickly identify when it fails to predict correct intents.
    • Insight into prediction accuracy enables targeted improvements using the NLU Workbench, improving user experience and Virtual Agent effectiveness.
    • The metrics and visualizations help justify adjustments to models and measure the impact of changes over time.
    • Transitioning to the new Platform Analytics dashboard ensures compliance and access to updated analytics capabilities.

    Virtual Agent (VA) uses the Natural Language Understanding (NLU) service to understand user input. Use the NLU Prediction tab to see how well NLU predicts intents, and to improve the intents so NLU makes better predictions.

    Important:

    Conversational Analytics dashboard is being prepared for future deprecation. It will be supported until deprecation but will no longer be available for installation. A new Conversational Analytics dashboard in Platform Analytics experience, which meets the compliance requirements of Government Community Cloud (GCC), and thus FedRAMP authorized, is available. See Conversational Analytics dashboard in Platform Analytics experience.

    For details on the deprecation process, see the Deprecation Process [KB0867184] article in the Now Support Knowledge Base.

    If you are an existing user of this dashboard and want to migrate analytics data to the new dashboard, see Migrate data to Conversational Analytics dashboard in Platform Analytics experience [KB1651556].

    The NLU Prediction tab on the Conversational Analytics Dashboard shows how well NLU is understanding user input in chat conversations. Virtual Agent comes with NLU models, but you can use the Activate the NLU Workbench to modify or create new models.

    NLU Predictions page.

    To use the NLU Prediction tab, you must have the Chat Analytics Viewer (chat_analytics_viewer) role.

    NLU Prediction graph

    The graph shows the daily performance of NLU predictions. The graph can show up to three lines, which represent:
    • Correct Predictions—NLU predicted multiple intents and showed them to users. They selected one.
    • Incorrect Predictions—NLU predicted multiple intents and showed them to users. They indicated that none of them were what they were looking for.
    • Auto Selected Predictions—NLU predicted a single intent based on chat input from the user. Sometimes, NLU returns multiple topics, each from a separate intent, and the user selects one.

    The example graph only shows Auto Selected Predictions. On 2021-01-27, one intent was auto-selected, and on 2021-01-31, two intents were auto-selected.

    Note:
    The vertical axis can have duplicate values because the axis has 10 increments. If the number of intents is less than 10, some values on the vertical axis appear more than once. For example, in the previous image, the maximum number of intents of all the days in the date range is 5. So, on the vertical axis there are two 1s, two 2s, two 3s, and so forth.
    How are these values determined? The dashboard collects the following data:
    1. Did the NLU prediction model determine an intent?
    2. Did the predicted intent match the topic bound in the model to the intent?
    3. What's the prediction score and did the dashboard show multiple options to the end user to specify the correct intent, or did the dashboard just display the topic associated with one intent.
    4. After showing the user multiple intents, did the user select one?
    The Conversational Dashboard uses the following algorithms on the data to populate the intent lists:
    • Correct Predictions—1, 2, 3, and 4 are true.
    • Incorrect Predictions—4 was false.
    • Auto Selected Predictions—3 was auto-selected, and there was no 4.

    It's possible for intents to appear in several categories. For example, the Activate Account (Ben) intent appears in the Correct Predictions and the Auto Selected Predictions columns because NLU correctly predicted the intent and only presented a single response to the user.

    Improving NLU predictions

    Clicking anywhere on the graph opens the Model Performance page. It shows a summary of intents predicted or not predicted over the date range specified on the graph.

    NLU Model Performance page.

    See Activate the NLU Workbench to see how to use the NLU Workbench to improve NLU predictions.