Performance enhancements for Indicator nightly job

  • Release version: Yokohama
  • Updated January 30, 2025
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Performance enhancements for Indicator nightly job

    The Yokohama release introduces significant performance enhancements for the Indicator nightly job in ServiceNow Governance, Risk, and Compliance (GRC). These improvements focus on parallel processing and optimized data handling, enabling faster and more efficient execution of a large number of indicators.

    Show full answer Show less

    Key Features

    • New Parallel Processing Architecture: Two additional custom queues have been introduced:
      • Indicator Data Queue: Processes the indicators themselves.
      • Supporting Data Queue: Handles events related to control, risk, and issue updates and collects supporting data.
    • Three-Step Processing Workflow:
      1. Indicators are queued in the Batch Indicator Data Queue, processed by the Batch Indicator Data Processor which executes indicators, sets status, and creates results or tasks.
      2. Processed indicators move to the Supporting Data Queue where the Indicator Supporting Data Processor updates control/risk statuses and manages issues according to configuration.
      3. The Supporting Data Collection Job collects supporting data from the Indicator Statistics Table using both default and custom queues, optimizing throughput.
    • Business Rules Optimization: Indicator result business rules are deferred during the job run and instead executed upon completion of indicator tasks to improve performance.
    • Queue Management: The Supporting Data Collection Job assigns batches in a round-robin manner to default and custom queues, ensuring efficient batch processing based on queue availability.
    • Data Model Enhancements:
      • Sample Collection Type Field: Added to the Indicator template form with options Count (default) or Percentage to specify how supporting data is collected.
      • Due Date Duration Field: Added to Indicator templates and forms to define manual indicator task due dates, triggering reminder emails for task owners.
      • JSON Field in Supporting Data Table: Introduced for efficient data management of indicator supporting data.

    Key Outcomes

    • Improved performance and scalability of indicator processing, enabling the execution of a large volume of indicators in significantly less time.
    • Enhanced data handling and update mechanisms for controls, risks, and issues related to indicators.
    • More efficient batch processing through intelligent queue management, reducing delays and resource contention.
    • Greater flexibility in how supporting data is collected and managed, supporting diverse indicator measurement needs.
    • Improved task management with due date tracking and reminders, helping ensure timely completion of indicator-related tasks.

    To support parallel processing capabilities, two additional custom queues such as the Indicator Data Queue for processing indicators, and the Supporting Data Queue for handling events related to control, risk, and issue updates and to collect the supporting data, have been introduced.

    New architecture for effective processing of indicators

    The execution of indicators has been optimized by implementing a three-step process, utilizing two custom queues for efficient processing.

    In the initial step, a set of indicators requiring processing is queued up in the Batch Indicator Data Queue. The Batch Indicator Data Processor then runs the indicators, sets the status, and creates the indicator results or tasks. The indicators are then moved to another custom queue for supporting data collection and updating dependent objects like controls, risks, and issues.

    In the second step, the Indicator Supporting Data Processor picks up the indicator records from the Indicator Supporting Data Queue and updates the control or risk status, also creating or updating issues based on the configuration. Then the indicator is pushed to the Indicator Statistics Table for support data collection.

    Finally, the Supporting Data Collection Job picks up the indicator records from the Supporting Data Statistics Table, starts collecting supporting data for the indicators, and leverages both the default queue and additional custom queues for efficient processing.

    Note:
    Indicator result business rules are not executed during the indicator job run. However, the same business rules are used while completing the indicator task. The indicator business rules are moved to the indicator job for achieving better performance during the execution.
    Figure 1. Processing indicator jobs using additional queues
    Workflow illustrating the processing of indicator jobs using additional queues

    This architecture supports execution of a large number of indicators in significant time. The Supporting Data Collection Job introduced to collect supporting data improves the data handling process considerably. Moreover, the time taken to execute the data is considerably less.

    Note:
    The Supporting Data Collection Job assigns the batches of indicator records to the default queue initially for supporting data collection. Depending on the availability of the batch processing queue and supporting data queue, the scheduled job assigns the subsequent batches of indicator records to these queues in a round-robin fashion, that is, only if these queues are free to process them.

    Data model changes

    The Sample collection type field added in the Indicator template form, has options either Count or Percentage. The default value is Count. If you select Percentage, you can provide the percentage value to collect the supporting data.

    The Due date duration (days) field is added for manual indicator in the Indicator template and Indicator forms to capture the indicator task due date. Based on this due date, reminder emails are sent to the indicator task owners.

    A new field introduced in the indicator supporting data table, which is of type JSON, helps in efficient data management.