---
sourceDocument: Australia Enable AI
sourceDocumentLink: https://servicenow-prod.fluidtopics.net/r/pt-BR/intelligent-experiences

 Release :

    - australia

ft:locale :

    - pt-BR

ft:publication_title :

    - Australia Enable AI

ft:clusterId :

    - platai

bundleId :

    - platai

workflow :

    - Platform


---

# Evaluation flow for batch evaluations

# Evaluation flow for batch evaluations {#ariaid-title1}

* Versão de lançamento: Australia
* 
* Atualizado 12 de mar. de 2026
* 
* ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 4 min. de leitura

Batch evaluation enables Eval admins to evaluate up to 100 completed virtual agent conversations at once, based on a saved query.
Flow name: Execute Batch Evaluation.

The flow creates evaluation records and invokes Now Assist skills for each eligible conversation, mirroring the single-conversation evaluation logic, but at scale. It enforces HR scope exclusions, topic/category validation, transcript construction rules, early
live-agent exclusions, and asynchronous scoring through skills.

Batch evaluations are performed using the following logic:  
Trigger

* Table: Evaluation set \[sn_na_conv_eval_evaluation_set\]
* Condition: State changes to In Progress and Evaluation type = Conversation
{#ai-eval-flow-batch__ul_pbd_hrl_ngc}  
Inputs

* Evaluation Set record with:
  * Query filter: A query that targets conversations to be evaluated (for example, sys_cs_conversation filters).
  * Evaluation type: Conversation
  * State: In Progress (to start)
  {#ai-eval-flow-batch__ul_hvp_lrl_ngc}
* LLM/Skills: Chat Topic Classifier, plus the evaluation skills listed after this.
{#ai-eval-flow-batch__ul_qkj_krl_ngc}  
High-level behavior

* Reads the query filter and randomly samples up to 100 conversations.
* Skips already-evaluated conversations.
* Excludes HR-scoped interactions.
* Uses Chat Topic Classifier to validate evaluation eligibility and extracts Topic and Category.
* Builds a transcript with controlled inclusion of Knowledge articles and catalog sources, and applies early live agent exclusions.
* Creates an Evaluation record and asynchronously invokes all selected evaluation skills, writing scores and rationale to metrics.
{#ai-eval-flow-batch__ul_pfc_trl_ngc}

Sequence of execution:  
Action 1: If the query filter isn't empty

* Purpose: Guard clause.
* Logic: Look up the Evaluation Set record and check the query filter field.
* If the query filter is present: Proceed to Action 2.
* If empty: Stop and optionally log No query provided.
{#ai-eval-flow-batch__ul_b45_msl_ngc}  
Action 2: Randomize conversations

* Purpose: Select a bounded, random sample of conversations from the provided query.
* Logic:
  * Execute the query to get matching conversation records.
  * Randomly select up to 100 conversations.
    * If \>100 matches, cap at 100.
    * If \<100, select all.
    {#ai-eval-flow-batch__ul_f4j_xsl_ngc}
  * Validate the query; if invalid, return false and an empty or partial array.
  {#ai-eval-flow-batch__ul_lnd_wsl_ngc}
* Outputs:
  * success: true/false
  * conversation_ids: array of sys_ids (max 100)
  {#ai-eval-flow-batch__ul_vyz_ctl_ngc}
* If success = true: Proceed to Action 3; otherwise, stop and log the validation error.
{#ai-eval-flow-batch__ul_xwk_tsl_ngc}  
Action 3: Look up the evaluation table to check prior evaluation

* Purpose: Avoid duplicate evaluations.
* Logic: For each conversation sys_id, check sn_na_conv_eval_evaluation for existing records indicating that it's already evaluated or is in progress (implementation choice: state not in canceled/failed).
* If not previously evaluated: Proceed to Action 4 for that conversation.
* If already evaluated: Skip this conversation, optionally log Already evaluated.
{#ai-eval-flow-batch__ul_pls_j1m_ngc}  
Action 4: Look up the interaction record

* Purpose: Enforce HR scope exclusion.
* Logic: Resolve the interaction related to the conversation. If its application scope contains hr, skip the conversation.
* If the scope doesn't contain hr: Proceed to Action 5.
{#ai-eval-flow-batch__ul_zxy_41m_ngc}  
Action 5: buildTranscript

* Purpose: Construct the final, minute-level transcript and determine downstream skill set and guardrails.
* Steps:
  * Aggregate all conversation messages.
  * Tag user messages as \[User\]: and virtual agent messages as \[Virtual Agent\]:.
  * Knowledge articles:
    * If genius results reference Knowledge articles, query the Knowledge article and replace the genius snippet with the entire article body.
    * Annotate with \[Virtual Agent\]: Help articles for user query: and wrap content within Article_Start and Article_End.
    * Constraints:
      * If the KB is HR-scoped or inaccessible, don't evaluate (skip conversation).
      * Truncate the article body to a maximum of 10,000 words.
      * If the KB content source is attached files (PDF/Word/Txt), fall back to the genius result instead of full file content.
      {#ai-eval-flow-batch__ul_tyy_rbm_ngc}
    {#ai-eval-flow-batch__ul_n3q_nbm_ngc}
  * Catalog Items:
    * If genius results reference catalog items, query sc_cat_item and build a string: catalog name, short description, description.
    * Annotate with \[Virtual Agent\]: Please choose one of the below options: and include citation order.
    {#ai-eval-flow-batch__ul_nh4_zbm_ngc}
  * Live Agent Exclusions:
    * If the first user message requests a live agent, skip evaluation.
    * If a live agent is invoked within the first 120 words, skip evaluation.
    {#ai-eval-flow-batch__ul_yph_2cm_ngc}
  {#ai-eval-flow-batch__ul_s2y_3bm_ngc}
* Outputs:
  * ExecuteEvaluation: true/false (post-guardrail outcome)
  * Chat transcript
  * Knowledge articles referred
  * Catalog items referred
  * First live agent occurrence: Sys_id of the conversation message (if present)
  * Skills to invoke:
    * Coherence Chat Evaluation
    * Conciseness Chat Eval
    * Context Retention
    * Inadequate Slot Filling Chat Eval
    * Intent Accuracy Chat Eval
    * Smooth Flowing Conversation Chat Eval
    * Truthfulness Hallucination Chat Eval
    {#ai-eval-flow-batch__ul_wtx_lpm_ngc}
  * Additional logs
  {#ai-eval-flow-batch__ul_z33_ggm_ngc}
* If ExecuteEvaluation = true: Proceed to Action 7; otherwise, skip the conversation.
{#ai-eval-flow-batch__ul_tkd_fbm_ngc}  
Action 6: If Block

* Purpose: Branch to record creation.
* Logic: If ExecuteEvaluation from Action 6 is true, go to Action 8.
{#ai-eval-flow-batch__ul_vzs_spm_ngc}  
Action 7: Chat Classifier Eval

* Purpose: Validate whether the conversation should be evaluated and extract high-level labels.
* Logic:
  * Build a lightweight transcript from sys_cs_message for classification input.
  * Invoke Chat topic classifier skill with the transcript.
  * Receive:
    * Execute evaluation: true/false
    * Topic Name
    * Category: IT or HR
    {#ai-eval-flow-batch__ul_mnn_1bm_ngc}
  {#ai-eval-flow-batch__ul_in2_x1m_ngc}
* If Execute evaluation = true: Proceed to Action 6.
* If false: Skip conversation and log the classifier decision.
{#ai-eval-flow-batch__ul_vr1_v1m_ngc}  
Action 8: Create or Update evaluation record

* Purpose: Persist an evaluation entry for this conversation.
* Table: sn_na_conv_eval_evaluation
* Field population:
  * Document conversation: Conversation reference
  * State: processing
  * Topic: from Action 5
  * Category: from Action 5
  * KB Referred: from Action 6
  * Catalog Referred: from Action 6
  * First live agent occurrence: from Action 6
  * Type: chat summarization
  * User: initiating user for the conversation
  * Message log: Additional logs from Action 6
  {#ai-eval-flow-batch__ul_j4m_ypm_ngc}
* On success: Proceed to Action 9.
{#ai-eval-flow-batch__ul_s5y_vpm_ngc}  
Action 9: For Loop over skills

* Purpose: Execute each selected evaluation skill.
* For each skill in the list from Action 6:
  * Action 10: invokeApiDefinition
    * Inputs: Skill Name, Conversation, Transcript, Evaluation Id
    * Behavior:
      * Invoke the Now Assist skill asynchronously.
      * The post-processor writes results into sys_generative_ai_response_validator.
      * Extract JSON response fields:
        * Score
        * Reason for Score
        * Examples supporting the reasoning
        {#ai-eval-flow-batch__ul_rph_mrm_ngc}
      * Create child metric records in sn_na_conv_eval_evaluation_metrics linked to the parent evaluation.
      {#ai-eval-flow-batch__ul_ins_3rm_ngc}
    {#ai-eval-flow-batch__ul_bmm_hrm_ngc}
  * Action 11: WaitPause seven seconds before proceeding to the next skill to manage rate limits or throttling.

  {#ai-eval-flow-batch__ul_dgc_2rm_ngc}
{#ai-eval-flow-batch__ul_wmf_brm_ngc}