Test and publish your model
Summarize
Summary of Test and publish your model
This feature enables ServiceNow customers to assess and improve the performance of their Natural Language Understanding (NLU) models by testing them against default or custom test sets. After testing, customers can publish the trained model to make it available for integration with applications such as Virtual Agent.
Show less
Testing Your Model
- Testing is performed from the NLU Workbench > Models section by selecting the model and accessing the Test and publish your model card.
- Test results include detailed statistics on how the model predicted intents for each utterance, categorized as Correct, Correct among multiple, Missed, and Incorrect predictions.
- Understanding these categories helps identify areas for improvement in the model’s intent recognition and confidence threshold settings.
- Testing requires the Multi-model Batch Testing feature, available through the NLU Workbench - Advanced Features application from the ServiceNow Store.
Interpreting Test Results
- Correct: Utterances where the model accurately predicted the intended intent, including correctly predicting no intent for irrelevant utterances.
- Correct among multiple: Cases with multiple predicted intents where the correct intent is included but accompanied by incorrect ones.
- Missed: Utterances where the model failed to predict an expected intent.
- Incorrect: Predictions where the model assigned an intent that was not correct for the utterance.
- Adjusting the model’s confidence threshold influences how confidently intents must be predicted to be accepted.
Publishing Your Model
Once the model is trained and tested, the Publish model button allows you to deploy the current version for use by other ServiceNow applications, such as Virtual Agent. Note that untrained models cannot be published and must be trained first.
Multi-model Batch Testing
This advanced feature lets you test multiple models simultaneously and against various test sets, providing a broader view of model performance. Access it via NLU Workbench > NLU Advanced Features > Multi-model Batch Testing. This is useful for comprehensive evaluation and managing multiple models efficiently.
Assess the performance of your NLU model to identify areas for improvement. Then publish your model to make it available to other applications such as Virtual Agent.
Summary usage
Test your Virtual Agent or AI Search model against its default test set to see how the model responds. Test results provide information you can use to improve your model.
To test your model, navigate to . Select the tab for your model's application, then select the name of the model.
In the Test and publish your model card, select View
phase.
Overview of testing and publishing your model
The Test and publish your model phase opens in the Overview page by default. Buttons for Run new test and Publish model are located here.
Overview provides information about a previous test run, with bar charts summarizing the test results.
If you have earlier test runs, you can view those by selecting from the Test run date list.
To drill down into the test results table, select the Detailed results tab. Each test utterance is listed in Detailed results, with its prediction.
Understanding test results
The test results show how your model responded to the utterances in the test set.
| Percentage | Description |
|---|---|
| Correct | The percentage of utterances for which your model correctly predicted the intent. When the model predicts no intent for utterances marked as Not relevant, that result is counted as Correct. |
| Correct among multiple | For utterances that had more than one intent predicted. The percentage of utterances for which the model correctly predicted the intent or intents, but also predicted intents that did not belong to the utterance. |
| Missed | The percentage of utterances for which your model did not predict an intent, even though there was an expected intent. |
| Incorrect | The percentage of utterances for which your model predicted an intent that was not correct. |
Testing can affect the model's confidence threshold. The confidence threshold determines how confident a model must be to predict an intent for an utterance. For more information on confidence thresholds, see NLU model settings.
For information about utterances which should not have any intent predicted, see Irrelevance detection in NLU.
Publish model
For more information on publishing your model, see Publish your NLU model.
Multi-model Batch Testing
In the Test and publish your model phase, you test your model against its default test set. With Multi-model Batch Testing, you can test against other test sets, test multiple models at once, and see your test results. To use Multi-model Batch Testing, navigate to .
For more information, see Multi-model Batch Testing.
For information about the process of testing, see Test your model.