Testing LLM topics

  • Release version: Xanadu
  • Updated February 26, 2025
  • 9 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Testing LLM topics

    This content explains how to preview, test, and debug Virtual Agent topics that utilize large language models (LLMs) within ServiceNow’s Virtual Agent Designer. Testing can be done directly in the chat test window, which defaults to the Service Portal chat client, or in third-party messaging apps where Virtual Agent is deployed. LLM topics offer faster and easier testing compared to Natural Language Understanding (NLU) topics because they do not require retraining after updating utterances.

    Show full answer Show less

    Testing supports multiple deployment channels including Now Assist panel, Microsoft Teams, and Slack when configured, allowing customers to test topics in their desired environments.

    Testing Methods and Options

    • LLM topics are tested via the Test options on the Virtual Agent Designer canvas or the Topics page.
    • When testing a single topic, the "Include topic discovery" option must be selected for assistant-related functionality.
    • If multiple LLM assistants are associated with a topic, you can select which assistant to test using the Assistant drop-down list.
    • Incomplete or problematic topics show badges and warnings; testing such topics opens an Issues window detailing each problem with links to fix them.
    • Active (published) topics can be tested via the Topics page with filtering by LLM assistant and options to include topic discovery.
    • The chat test window supports user feedback on LLM responses with thumbs-up and thumbs-down icons for each utterance.

    Chat Test Window Features

    The chat test window provides several tabs for detailed insights during testing:

    • Analyze test phrases: Displays results for topic discovery, including matching skills, Knowledge Base articles, and catalog items, leveraging Genius Results and Semantic Search if enabled.
    • Modify instructions: Shows LLM user input nodes and their instructions, allowing edits to optimize conversation flow. It also highlights enforced prompts and warnings for potentially offensive content.
    • Variables: Lists all conversation variables (input, script, Live Agent, and passed variables) and their current values to track conversation state.
    • Edit variables: Available when testing topic blocks and custom controls to modify variable values.
    • Context: Enables setting context variables that influence topic intent or live agent routing; applicable for topics, setup topics, and small talk when topic discovery is included.
    • Logs: Displays processing details for the conversation.

    Practical Guidance for ServiceNow Customers

    • Ensure the Now Assist Topics skill is activated to access the updated UI and enhanced testing features.
    • Use the Assistant drop-down to select the relevant LLM assistant when multiple assistants exist, ensuring accurate testing.
    • Respond to issues indicated by badges before testing for smoother debugging and topic refinement.
    • Leverage feedback icons on utterances to gather insights on conversation quality and improve topic performance.
    • Use the Analyze test phrases tab to review semantic search and skill discovery results, adjusting topic descriptions or instructions to improve confidence scores.
    • Modify instructions directly in the testing interface to optimize how LLM interprets user input and conversation flow without needing to republish immediately.
    • Set context variables during testing to simulate different user environments or routing scenarios.

    Next Steps

    After testing, close the chat test window and apply insights gained to fine-tune your topic. Address low confidence ratings by updating topic metadata or instructions in LLM user input nodes to enhance accuracy and user experience.

    Preview, test, and debug topics that use large language models (LLMs) in the Virtual Agent Designer chat window.

    You can test your topic's functions by running your conversation in a chat test window as you work. The web (Service Portal) chat client is the default test window.

    Large language model (LLM) topics are faster and easier to test compared to Natural Language Understanding (NLU) topics. For example, you don't need to retest your topic after updating utterances and retraining models as with NLU topics.

    If you're using the Virtual Agent integrations with third-party messaging apps, elements in your conversation might appear differently in third-party messaging applications. Test your conversations in any third-party applications where you want to deploy Virtual Agent.

    Note:
    If the Now Assist panel, Microsoft Teams application, or Slack application is configured for your environment, preview options for those channels are displayed in the Test button list. Select Preview in Now Assist panel or Preview in Microsoft Teams in the list to test your topic in those environments.

    Testing your LLM topic in the chat test window

    Figure 1. LLM test options
    Note:
    An updated Virtual Agent Designer user interface is available when you install Now Assist in Virtual Agent and turn on the Now Assist Topics skill. This content assumes that you have activated this skill and can see the list view. If this skill is not enabled, you will see the legacy UI and topics page. For more information, see Virtual Agent Designer legacy topics page.
    You can find LLM testing options on the Virtual Agent Designer canvas and the home page.
    Test LLM topics on the Virtual Agent Designer canvas by working with the Test options on the topic header bar. When testing from a single topic, the Include topic discovery option must be selected to work with the assistants. If the topic is associated with multiple LLM assistants, use the Assistant drop-down list to select which assistant you want to test. For more information about LLM assistants, see LLM assistants.
    Note:
    If your topic is missing any necessary information, an incomplete badge appears in the corner of the flow diagram pane. A yellow or red warning badge also appears next to each node. The incomplete badge lists the total number of issues, while the local warning badges show how many issues are found in each node. If you select Test when there are issues, an Issues window opens with the number of issues present along with the details of each issue. These details include a full description and a hyperlink to each incomplete item.

    Alternatively, you can test active (published) LLM topics on the Topics page. Use the LLM Assistant filter to restrict your topics to only topics associated with that assistant, and then select Test active topics. If you don't use the LLM Assistant filter, select Test LLM assistant topics from the Test active topics drop-down list. When using the Test active topics option or sub-options from the Topics page, topic discovery is enabled, so it's not listed as an option. When testing from the Topics page, the Assistant drop-down list appears in the chat test window. You must have previously established at least the default Now Assist in Virtual Agent assistant to see the Assistant drop-down list. When a topic is associated with just one LLM assistant, the Assistant drop-down list defaults to that assistant name. If you have multiple assistants, select which assistant you want to work with using the Assistant drop-down list. For more information about creating multiple LLM assistants, see Manage LLM virtual agents on the Assistants screen.

    The chat test window opens in the Now Assist in Virtual Agent chat widget. The Assistant drop-down list and Include topic discovery option might be displayed depending on where you originated the test from and if you have established any LLM assistants.

    You can provide feedback to each utterance made by the LLM by selecting from the like thumbs-up icon Thumbs up icon or dislike thumbs-down icon Thumbs down icon options that appear when you hover over an utterance. All LLM user inputs aside from the Input Collector have these feedback options.

    Figure 2. LLM response feedback options
    LLM response asking the user "Which color do you prefer: red or green?" followed by "red" or "green" choices. Like and dislike feedback options are shown in the message's corner.

    The chat test window also displays adjoining tabs that provide details about your topic as you test it. The following tabs are available when testing LLM topics:

    • Analyze test phrases - Results for topic discovery based on your input.
    • Modify instructions - List of all the instrucions in the topic that are sent to the LLM. This tab is only available when testing a single topic.
    • Variables - List of all the variables used in the conversation, such as input and Live Agent variables.
    • Edit variables - Options for editing the variables used in this topic.
    • Context - Options for specifying the context (using context variables) in which a topic is run.
    • Logs - List of the processing performed.
    Figure 3. Example test window and tabs
    Different tabs appear based on the type of LLM topic that you are testing.
    The following tabs appear when you test their related topic types on the Virtual Agent Designer canvas:
    • The Analyze test phrases, Modify instructions, Variables, and Logs tabs appear for all LLM topic types.
    • The Edit variables tab appears for topic blocks and custom controls.
    • The Context tab appears for topics, setup topics, or small talk topics when you select Include topic discovery.

    If you run tests from the Topics page, the test window shows only the Analyze test phrases, Variables, Context (available by default, with no Include topic discovery option), and Logs tabs.

    Analyze test phrases tab

    When you test LLM topics, results for topic discovery appear based on your input. When testing topics associated with a primary LLM assistant, only the primary assistant's promoted topics appear. When you input a test phrase, you can see a variety of search results when Genius Results are enabled including skill (topic) discovery, Knowledge Base (KB) articles, and catalog items. Under the skills search results, a Matching badge appears next to the skill discovered, while variables and values may also be listed (such as the variable @laptop_make and the value macbook), depending on the topic. These Genius Results only appear if you’re testing a published topic and have selected the Include topic discovery option. If the Include topic discovery option appears inactive, publish the topic and select an LLM assistant on the Properties tab to work with topic discovery. A Search indexing in progress message might appear, but you can still test while the search indexing runs although topic discovery might not be updated. For more information about how Genius Results work, see Now Assist in AI Search.

    Additionally, when testing active LLM assistant topics from the Topics page, you see skills results for Semantic search. Semantic search analyzes the meanings and context of your search terms and uses that information to find results with similar meanings. It improves search recall by interpreting natural language to more accurately reflect your search's intent. If semantic search is deactivated for topic discovery testing, those results are not displayed. For more information about semantic search, see Semantic vector search in AI Search.

    Figure 4. Example Analyze test phrases tab for LLM topics
    Analyze test phrases tab, showing test phrase results for Skill Discovery, with Matching result, and Input and Output Skills categories.

    Modify instructions tab

    If your topic contains any instructions that are sent to the LLM, they’re shown in the Nodes category. The Nodes category lists LLM user input nodes on the canvas, their field names, and their contents.

    You can modify the content of any of the instructions to develop optimal instructions for the conversation. If a node has Enforce user prompt active, a blue check icon VAD Advanced options checkmark and a User prompt enforced message appears. Deactivate Enforce user prompt to modify the node's instructions.

    If any of your user inputs have Allow automatic slot filling activated, the input's Detail Description becomes a static field. If Allow automatic slot filling is inactive, you can define detail description using a script or data pill picker. For more information, see the Allow automatic slot filling table entry in Text user input control or any other LLM user input controls.

    Each instruction's status starts at Original. If you alter an instruction's content, its status changes to Modified and a revert iconRevert icon appears next to it. The total number of instructions modified is listed at the bottom of the tab next to a Save to topic button. To test the modified instructions, select Apply and restart. To undo changes to an instruction, select the revert icon next to the instruction. Alternatively, undo all changes by selecting Revert that activates when you modify any instructions.
    Note:
    If you select Save to topic, all changes are saved and all Modified messages reset to Original.
    Figure 5. Example Modify instructions tab
    Modify instructions tab showing user information with LLM instruction guidelines link, and Nodes held within an Input collector, including a locked node with User prompt is enforced message.
    Figure 6. Example Modify instructions tab with a modification
    Modify instructions tab with Modified instruction and Save to Topic updates. The Revert and Apply and Restart options activate when you make any changes.
    If instructions in a node include potentially offensive content, a warning badge appears on the node. A yellow warning icon Yellow warning icon also appears next to the node name. Selecting Apply and restart inputs those instructions and removes the badge and icon, and the Modified status reverts to Original.
    Figure 7. Example Modify instructions tab with potentially offensive content warnings
    Modify instructions tab showing offensive content warning badge, icons, and hover message. Selecting Apply and restart applies the instructions and removes the warnings.

    Variables tab

    The Variables tab displays a list of all the variables used in the conversation and the associated values captured as the conversation progresses, so that you can follow along. A conversation can have these variable types:
    • Input variables
    • Script variables
    • Live Agent variables
    • Variables passed between a calling topic and topic block
    The list is separated into sections by variable types. The following example shows the Input variables section. Notice that for the static list control, both the display label and value are captured for the selected choice.
    Figure 8. Example list of input variables
    Variables tab that shows the input variables and values, next to a screen capture of the conversation flow.

    The following example shows the Input variables section for the grouped list control. This variable information appears similar to the static list control, but the variables are separated by each group of the grouped choice.

    Figure 9. Example list of grouped choice variables
    Variables tab separated by group, with the node and Group name highlighted.

    Edit Variables tab

    When testing topic blocks and custom controls, you can edit the variables found in the nodes.

    Edit variables in the topic's nodes.

    Context tab

    The Context tab appears when you’re testing topics, setup topics, or small talk topics, to specify a different context for the chat. Choose a context variable from the list. The variables contain contextual information that can be used to determine topic intent or control how chats are routed to live agents. For example, you could select portal from the list of variables and enter the portal name IT Express. The Context tab is unavailable when creating test cases.

    For more information about defining context variables, see Configure context variables for storing chat-related information. For more information about live agent variables that are included with Virtual Agent, see Live agent chat context variables.

    Figure 10. Example Context tab
    Context tab with the context portal value of service portal.

    Next steps

    When you're done testing your topic, close the test chat window. If necessary, use the test information to adjust your topic to perform more accurately. For example, the results on the Analyze test phrases tab may return low scores or Unsure or Unknown confidence ratings. Improve scores by updating the topic description or instructions in the LLM user input nodes.