--- sourceDocument: Xanadu ServiceNow AI Platform Administration sourceDocumentLink: https://servicenow-prod.fluidtopics.net/r/xanadu/platform-administration Release : - xanadu ft:locale : - en-US ft:publication_title : - Xanadu ServiceNow AI Platform Administration ft:clusterId : - platadm bundleId : - platadm workflow : - Platform --- # Search query parameter evaluation framework # Search query parameter evaluation framework {#ariaid-title1} * Release version: Xanadu * * Updated November 7, 2025 * * ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 8 minutes to read AI Search performs A/B test evaluations that compare result relevancy for alternate sets of search query parameter values. These evaluations determine the models that AI Search uses for machine learning relevancy and for Q\&A Genius Results. The search query parameter evaluation framework enables A/B relevancy testing on live search traffic. AI Search considers results from this live testing along with its offline evaluation of aggregated search signal data when making the following determinations: * Determining which relevancy model to publish for a search profile. For details on generation and publication of relevancy models, see [Machine learning relevancy in AI Search](https://servicenow-prod.fluidtopics.net/QecNLKIDvDfIHa9lGOi6~Q "AI Search displays the most relevant search results for a query first. Machine learning automatically tunes search result relevancy scoring for search experiences based on aggregated user interactions."). * Determining which machine reading comprehension (MRC) model to use when validating potential Q\&A Genius Result answers. For more information on the MRC model used for Q\&A Genius Results, see [Q\&A Genius Results](https://servicenow-prod.fluidtopics.net/IKsSWCrpxy1rzrXverZNPA "Q&A Genius Results display top search results extracted from HTML fields of records on the Knowledge [kb_knowledge] table and tables that extend it. Each Q&A Genius Result answer card shows a topic snippet and an answer snippet extracted from a single Knowledge article. You can view the full article directly from the answer card."). {#srch-qry-parm-eval-fw-ais__ul_j41_lq4_hzb} AI Search computes A/B testing evaluation results on a nightly basis. Note: The search query parameter evaluation framework does not support user configured A/B testing. All A/B testing is automatically performed by AI Search as part of its evaluation of machine learning relevancy models. The search query parameter evaluation framework is part of the Adaptive Telemetry suite of features. ## Search query parameter evaluation framework tables {#ariaid-title2} The search query parameter evaluation framework for AI Search uses tables to store results and metrics from A/B testing of live search traffic. AI Search uses the stored data when determining which relevancy model and Q\&A Genius Result answer validation model to publish for a search application. ### A/B Testing Evaluation table {#srch-qry-parm-eval-fw-tables-ais__section_svn_xlj_yxb} The A/B Testing Evaluation \[evaluation\] table stores details for A/B testing evaluations performed on live search traffic. Note: The system automatically cleans this table, removing records for inactive evaluations that are more than two years old. {#srch-qry-parm-eval-fw-tables-ais__section_svn_xlj_yxb__entry__2} | Column | Description | |-|-| | Number | Autogenerated number to identify the evaluation. | | Name | Name and description for the evaluation. | | Source | Name of the table containing records to run the evaluation for. | | Testing Scope | Scope of the evaluation. * Supported values: * ALL: Run the evaluation against all records in the table specified by Source. * SELECTED: Only run the evaluation against records specified in the Selected Artifacts field value. * Default value: ALL | | Artifact Provider | List of artifact filtering classes that provide records for testing. | | Selected Artifacts | Comma-separated list of record sys_id values. When Testing Scope is set to SELECTED, the evaluation only runs against records with sys_id values specified in this list. | | Minimum Signals for Evaluation | Minimum number of search signals the evaluation should collect before ending. * Type: integer * Default value: 0 (no minimum) * Maximum value: 30000 For details on how this condition interacts with the Minimum Days for Evaluation condition, see End Criteria Operator. | | Minimum Days for Evaluation | Minimum number of days the evaluation should be active before ending. * Type: integer * Default value: 0 (no minimum) * Maximum value: 180 For details on how this condition interacts with the Minimum Signals for Evaluation condition, see End Criteria Operator. | | End Criteria Operator | Logical operator to use for evaluation end conditions when both Minimum Signals for Evaluation and Minimum Days for Evaluation conditions exist. * Supported values: * AND: The evaluation remains active until the Minimum Signals for Evaluation and Minimum Days for Evaluation conditions are both satisfied. * OR: The evaluation remains active until either of the Minimum Signals for Evaluation or Minimum Days for Evaluation conditions is satisfied. * Default value: OR | [Table 1. evaluation] ### A/B Testing Evaluation Execution table {#srch-qry-parm-eval-fw-tables-ais__section_kct_xlj_yxb} The A/B Testing Evaluation Execution \[evaluation_execution\] table stores details of individual operations executed as part of A/B testing evaluations for live search traffic. {#srch-qry-parm-eval-fw-tables-ais__section_kct_xlj_yxb__entry__2} | Column | Description | |-|-| | Number | Autogenerated number to identify the evaluation operation. | | Evaluation | Reference to the record for the evaluation on the A/B Testing Evaluation \[evaluation\] table. | | Artifact ID | Sys_id for the record analyzed by this evaluation operation. | | State | State for the record evaluation execution. * Supported values: * Queued: The evaluation operation is queued for execution. * In Progress: The evaluation operation is underway. * Scoring: The evaluation operation is in the scoring phase. * Complete: The evaluation operation completed successfully. * Errored: The evaluation operation failed with an error. * Canceled: The evaluation operation was canceled by the system. * Suspended: The evaluation operation was suspended by the system. * Default value: Queued | | Start Date | Date and time when the evaluation operation started. | | End Date | Date and time when the evaluation operation ended. | [Table 2. evaluation_execution] ### A/B Testing Evaluation Parameter table {#srch-qry-parm-eval-fw-tables-ais__section_xx1_ylj_yxb} The A/B Testing Evaluation Parameter \[evaluation_parameter\] table stores details of individual search query parameters considered in A/B testing evaluations for live search traffic. {#srch-qry-parm-eval-fw-tables-ais__section_xx1_ylj_yxb__entry__2} | Column | Description | |-|-| | Evaluation | Reference to the record for the evaluation on the A/B Testing Evaluation \[evaluation\] table. | | Name | Name and description for the evaluation parameter. | | Use Artifact's Assigned Parameter | * Type: true \| false * Supported values: * true: Compare the specified Parameter Values to the currently assigned search query parameter value on the provided artifact. * false: Don't compare the specified Parameter Values to the currently assigned search query parameter value on the provided artifact * Default value: true | | Parameter Values Traffic Percentage | Percentage of search queries to apply the specified Parameter Values to during the A/B evaluation. * Type: integer * Default value: 0 * Maximum value: 100 | | Parameter Values | JSON list of values to compare to the current search query parameter value when Use Artifact's Assigned Parameter is true. | | Parameter Type | Type for the search query parameter. The selected value determines how the Parameter Values are used during and after an A/B evaluation. Supported values: * Search Context Parameters: Merge and override search profile context parameters with the Parameter Values for search query requests. When the evaluation ends, use the search query context parameter value set with the winning score to override or merge the relevant search profile context parameters. * Search QnA Genius Result Configuration: Apply the Parameter Values to Q\&A Genius Result configurations used in search query requests. When the evaluation ends, update the relevant search profile to include the Genius Result configuration parameter value with the winning score. * Search QnA Model: Apply the Parameter Values to Q\&A Genius Result models used in search query requests. When the evaluation ends, update the relevant search profile to include the Q\&A Genius Result model parameter value with the winning score. * Search Relevancy Model: Apply the Parameter Values as search relevancy models to use in search query requests. When the evaluation ends, update the relevant search profile to include the relevancy model parameter value with the winning score. * Search REST Parameters: Merge and override search profile query parameters with the Parameter Values for search query requests. When the evaluation ends, use the search query parameter value set with the winning score to override or merge the relevant search profile parameters. | | Score Calculation Type | Form of calculation used to compute search query parameter value scores and find the best value. Supported values: * Average Click Position: The best search query parameter value is the one with the lowest average click position score. Note: AI Search computes the average click position score by dividing the sum of all selected search result ranks by the number of searches. The highest-ranked search result has rank 1, the next-highest has rank 2, and so on. As an example, if you perform two searches, selecting the first result in one case and the second result in the other, your average click position score is (1 + 2) / 2 = 1.5. * Genius Result Helpfulness: The best search query parameter value is the one with the highest helpfulness score. This calculation takes into account whether a relevant Genius Result answer appeared in your search results. Note: The helpfulness score is a metric that indicates whether Genius Result answers were helpful in the context of your search. * Helpfulness: The best search query parameter value is the one with the highest helpfulness score. Unlike Genius Result Helpfulness, this calculation doesn't take into account whether a relevant Genius Result answer appeared in your search results. | | Signal Provider | The provider for search signals needed to score the search query parameter. Search Event Signal Provider is the only supported value. | [Table 3. evaluation_parameter] ### A/B Testing Evaluation Parameter Result table {#srch-qry-parm-eval-fw-tables-ais__section_q2g_ylj_yxb} The A/B Testing Evaluation Parameter Result \[evaluation_parameter_result\] table stores calculation results for individual search query parameters considered in A/B testing evaluations for live search traffic. {#srch-qry-parm-eval-fw-tables-ais__section_q2g_ylj_yxb__entry__2} | Column | Description | |-|-| | Evaluation Execution | Reference to the record for the evaluation execution on the A/B Testing Evaluation Execution \[evaluation_execution\] table. | | Parameter Evaluation | Reference to the record for the search query parameter on the A/B Testing Evaluation Parameter \[evaluation_parameter\] table. | | Best Value | Best value for the search query parameter, as determined by the Winning Score. | | Winning Score | Numeric score for the search query parameter, determined using the Score Calculation Type. | | Score Metadata | Metadata from the score computation for the search query parameter. | [Table 4. evaluation_parameter_result] ## Exclude a search profile from the search query parameter evaluation framework {#ariaid-title3} Exclude a search profile from A/B testing evaluations of live AI Search traffic. This procedure prevents AI Search from using A/B testing evaluation results when publishing the search profile's search result relevancy model and its Q\&A Genius Result answer validation model. ### Before you begin Role required: ais_admin ### About this task The search query parameter evaluation framework performs A/B testing evaluations of search configuration settings using live search traffic. By default, AI Search evaluates configuration settings for all search profiles. Search administrators can exclude individual search profiles from the search query parameter evaluation framework. Excluding a search profile from the framework prevents AI Search from performing A/B testing evaluations for live search traffic that uses the excluded search profile. Note: When you exclude a search profile from evaluations, AI Search no longer considers evaluation results when updating the machine learning relevancy and machine reading comprehension (MRC) models for that search profile. As a result, relevancy scoring settings and Q\&A Genius Result answer filtering settings for the search profile may be less reflective of your search traffic. For more information on how AI Search uses A/B evaluation testing results when publishing these models, see [Machine learning relevancy in AI Search](https://servicenow-prod.fluidtopics.net/QecNLKIDvDfIHa9lGOi6~Q "AI Search displays the most relevant search results for a query first. Machine learning automatically tunes search result relevancy scoring for search experiences based on aggregated user interactions.") and [Q\&A Genius Results](https://servicenow-prod.fluidtopics.net/IKsSWCrpxy1rzrXverZNPA "Q&A Genius Results display top search results extracted from HTML fields of records on the Knowledge [kb_knowledge] table and tables that extend it. Each Q&A Genius Result answer card shows a topic snippet and an answer snippet extracted from a single Knowledge article. You can view the full article directly from the answer card."). ### Procedure 1. Navigate to AllAI SearchSearch ExperienceSearch Profiles. 2. Open the record for the search profile that you want to exclude from A/B testing evaluations of live search traffic. 3. If the Search Profile form doesn't already display the Exclude from evaluation field, configure the form layout to make the field visible. For details on configuring a form layout to make fields visible, see [Configuring the form layout](https://servicenow-prod.fluidtopics.net/1phGIP6~lfwMUKKczOxN0Q#configure-form-layout "Administrators or users with the personalize_form role can configure the form and related list layout."). 4. Select the Exclude from evaluation option. 5. Select Update. ### Result AI Search no longer performs A/B testing evaluations for traffic that uses the excluded search profile. Machine learning relevancy no longer updates the relevancy model for the search profile. *[\>]: and then