Multi-model Batch Testing

  • Release version: Yokohama
  • Updated January 30, 2025
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Multi-model Batch Testing

    Multi-model Batch Testing enables ServiceNow customers to evaluate multiple Natural Language Understanding (NLU) models simultaneously by testing them against large sets of utterances. This capability helps assess model performance by comparing predicted intents against expected intents, facilitating model refinement and improvement.

    Show full answer Show less

    Key Features

    • Test Sets: Upload test sets in CSV or XLSX format containing utterances and their corresponding expected intents. Test sets can include up to 10,000 rows and should represent realistic user utterances in the same language as the models.
    • Intent Coverage: Test sets should cover at least 60% of the model’s intents for optimal results. Including utterances with no expected intent helps evaluate the model’s ability to recognize irrelevant inputs.
    • Multi-Model Testing: Run batch tests across multiple trained NLU models to compare their prediction accuracy and confidence levels.
    • Test Results Dashboard: Access a comprehensive Test results page displaying summaries, prediction percentages, and detailed breakdowns of missed or incorrect intents for each model tested.
    • Detailed Analysis: Drill down into individual utterance predictions, filter results, and export detailed test outcomes to CSV for further analysis.
    • NLU Workbench Integration: Available as part of the NLU Workbench - Advanced Features app from the ServiceNow Store. Requires activation of the corresponding plugin on the ServiceNow instance.

    Practical Use and Benefits

    By using Multi-model Batch Testing, ServiceNow customers can efficiently validate and compare multiple NLU models to ensure they meet accuracy requirements before deployment. This process helps identify intents needing improvement, supports continuous model tuning, and ultimately enhances the quality of user interactions handled by conversational AI.

    Getting Started

    • Ensure the NLU Workbench - Advanced Features plugin is activated.
    • Create test sets with representative utterances and expected intents aligned to your models’ languages.
    • Run multi-model batch tests via the interface to generate performance reports.
    • Review detailed results to identify and address misclassifications.

    Test multiple Natural Language Understanding (NLU) models against a large set of utterances to evaluate the performance of the models. Add test sets, test multiple models, and see test results.

    Summary usage

    Use Multi-model Batch Testing to create and upload test sets comprised of utterances and their expected intents. You can then run tests against your NLU models.

    Multi-model Batch Testing works with models for all supported NLU languages. See NLU language support.

    Installation

    Multi-model Batch Testing is part of the NLU Workbench - Advanced Features app available on the ServiceNow® Store.

    To use Multi-model Batch Testing, ensure that the NLU Workbench - Advanced Features (com.snc.nlu.workbench.advanced) plugin is active on your instance. For more information, see Install NLU Workbench - Advanced Features and Activate the NLU Workbench.

    Test sets

    Test sets are lists of utterances and matched intents. Create a test set by using a table in a CSV or XLSX (Excel workbook) file. The table should contain two columns: one for utterances, and one for the expected intent. Your test set can include up to 10,000 rows.

    To get the most out of testing your NLU models, your test sets should include utterances that the model is likely to encounter from your users. Test utterances should be in the same language as the model to be tested. The test set should also include utterances with no expected intents. Including utterances with no expected intent helps assess your model's ability to detect utterances which are irrelevant and shouldn't have any intent predicted.

    By including these types of utterances, the test better assesses the model's ability to perceive intents and respond to your users. If your test set does not cover at least 60% of the intents of the models, you can still run the test but the recommended threshold may not be optimal.
    Note:
    Certain test utterances are skipped during the test if their expected intent does not match any intents in the models.

    To create a test set, see Create a test set.

    After you have a test set, you can test trained NLU models. To begin testing, see Run a multi-model batch test.

    After running a test, your results appear on the Test results page.

    Test results

    The Test results page lists your completed and in-progress tests. At a glance, the results page shows the models tested against, the number of utterances, and prediction percentages.

    Multi-model Batch Testing page with completed tests.

    To see the details of a test result, click the name of the test set.

    The Overview page shows summary information about the results and includes a graphic with a breakdown of predictions.

    The Intents that need attention (Current model) shows the top 5 missed and incorrect intents. Click the intent name to drill down into the test utterances that were predicted incorrectly. Use this information to improve the model.

    The Detailed results tab lists information about each utterance that was tested. From here, you can see the prediction outcome and confidence per model for each utterance. Filter the results by using the search bar or interacting with the filter tools and column headers.

    You can also export the test results to a CSV file by clicking Export. The file includes the same columns as the detailed results page.

    For more information on understanding your test results, see Test and publish your model.