Scheduled Jobs

• CE Populate Value Aggregates Chats – Daily: Runs daily to randomly select 1000 conversations from the previous day, classifies chat durations (small, medium, large), identifies if Knowledge articles or catalog items were used, evaluates chat performance, and populates aggregated data into the Evaluation Value Aggregates table.
• Evaluation Value Calculation - Runs Only Once After Install: Clears the Evaluation Value Aggregates table and recalculates aggregated values based on the initial evaluation date.

Tables

Several tables support evaluation data management:

• Evaluation, Evaluation Configurations, Evaluation Metrics, Evaluation Set, and Evaluation Value Aggregates for storing various evaluation data.
• Remote tables like Conversation Evaluator Value Calculations and Conversation Weekly Calculations compute and provide time savings and efficiency percentages by chat size and Knowledge article/catalog usage over selected date ranges.

System Properties

System properties allow customization and control of evaluation parameters:

• Minimum records for error band calculation
• Weights for evaluation metrics to compute composite scores
• Maximum daily evaluation record count and total sampled conversations
• Definitions for classifying conversations by inbound message count (small, medium, large)
• Weights for value calculations by chat type
• Control for rerunning value calculations post-installation

Business Rules

Business rules automate updates and scoring related to evaluation metrics, including:

• Adding info messages when an Evaluation set state changes
• Scaling and updating labeling metric scores
• Updating deviation scores for LLM-generated metrics
• Calculating composite scores when evaluations complete

Flows and Flow Actions

• Execute Evaluation flow: Evaluates conversations upon completion (default deactivated; can be activated to evaluate chats in real-time).
• Execute Batch Evaluation flow: Performs batch evaluation of up to 100 completed conversations when an Evaluation set is created or updated.
• Flow actions support operations like randomizing conversations, invoking LLM capabilities, classifying chats, building transcripts, and checking evaluation conditions.

Script Includes

• evalExecuteCondition: Updates evaluation conditions to ensure quality.
• evalUtils: Provides utility functions used across the Evaluation processes.

What This Enables for ServiceNow Customers

With these components, ServiceNow customers can automate the evaluation of virtual agent conversations, gain insights into chat performance, and measure efficiency improvements, particularly when Knowledge articles or catalog items are used. The system properties and business rules allow customization to align evaluations with organizational standards. Scheduled jobs and flows provide flexibility for real-time or batch processing, ensuring scalable and timely assessments.