Archive rule and destroy rule properties

  • Release version: Xanadu
  • Updated August 1, 2024
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Archive rule and destroy rule properties

    The archive process in ServiceNow uses an automated producer and consumer model to efficiently manage system resources during data archiving. Records to be archived are divided into chunks and processed in batches. System properties control the behavior of this process, allowing customers to fine-tune performance and resource usage based on their needs.

    Show full answer Show less

    Key Features

    • Producer-Consumer Model: The ArchiveProducerJob creates chunks of records (sysids) grouped by archive rule in the Archive Job Execution Chunks [sysarchiverunchunk] table. Consumers (ArchiverJob workers) process these chunks.
    • Chunk Size and Limits: The number of records per chunk is controlled by glide.db.archive.chunksize, and the total number of records processed per rule across all chunks is limited by glide.db.archive.max.rule.records.
    • Consumer Workers: By default, four consumer workers process archive chunks concurrently, controlled by glide.db.archiving.maxconsumerworkers. Each consumer processes a maximum number of chunks specified by glide.db.archive.max.batches, then shuts down and schedules a new consumer.
    • Chunk Processing Timeout: If a chunk remains active longer than the timeout defined in glide.db.archive.chunk.max.process.time, it is marked as an error, indicating a possible abrupt termination of the consumer. This does not stop other consumers.
    • Debug Logging: Verbose logging of the archiving process can be enabled via glide.db.archive.debug to assist in troubleshooting and monitoring.
    • Manual Configuration: Customers can adjust all these properties to control the archive and destroy process behavior to better fit their system load and archiving requirements.

    Key Outcomes

    • Improved control over the archiving process to prevent excessive resource consumption.
    • Ability to customize batch sizes, concurrency, and processing timeouts to optimize performance.
    • Automated handling of stuck or failed chunks without interrupting ongoing archiving jobs.
    • Enhanced monitoring capabilities through optional debug logging.
    • Ensures the archiving process is scalable and reliable across different ServiceNow instances and workloads.

    Rules to prevent the archive process from consuming too many system resources have been automated. You can manually change the system behavior.

    The archive process uses a producer and consumer model. Records are queued for archiving in the Archive Job Execution Chunks [sys_archive_run_chunk] table in batches.

    • The ArchiveProducerJob creates chunks of sys_ids records that need to be archived. Each chunk contains records pertaining to a single archive rule. The maximum number of records in a chunk is specified by the property glide.db.archive.chunk_size. The maximum number of records for a single rule across all chunks is specified by glide.db.archive.max.rule.records.
    • The producer starts up 4 (default) ArchiverJob consumers. The exact number of consumers is specified by glide.db.archiving.max_consumer_workers.
    • Consumers read and process chunks of sys_ids. The number of chunks each consumer processes is specified by glide.db.archive.max.batches. The chunks processed by each consumer may be from different archive rules.
    • Consumers repeat reading chunks of sys_ids until none remain or the limit of chunks a consumer can process (specified by glide.db.archive.max.batches) is reached, after which it shuts down and schedules another consumer.
    • If a chunk remains in the running state longer than what's specified in glide.db.archive.chunk.max.process.time, the system assumes that the consumer abruptly terminated, and it changes the chunk’s status to error. This doesn't stop or end any running consumers.
    Note:
    The archive and destroy properties listed in the table are not used if the Archive Job Execution Chunks [sys_archive_run_chunk] table is deleted.
    To manually control how many records the archive rule and destroy processes at one interval, you can change these default settings by adding the following system properties.
    Table 1. Archive System Properties
    Name Description Type Default Value
    glide.db.archive.chunk_size Number of unique identifiers added to each record in the Archive Job Execution Chunks [sys_archive_run_chunk] table. The record is formatted as CSV values in the keys field of the Archive Job Execution Chunk (sys_archive_run_chunk.do) form. Integer 1000
    glide.db.archive.chunk.max.process.time Number of milliseconds a record can remain in the Active state in the Archive Job Execution Chunks [sys_archive_run_chunk] table. The time is the difference between the system time and the value of the Start field in the Archive Job Execution Chunks [sys_archive_run_chunk] table. Integer 600000
    glide.db.archive.max.rule.records Maximum number of records to batch per consumer, per rule. Integer 10000
    glide.db.archive.debug Option that enables verbose logging when copying the column definitions from the original table to the archive table. true | false false
    glide.db.archive.max.batches Number of chunks, which are records from the Archive Job Execution Chunks [sys_archive_run_chunk] table, processed by a consumer before it shuts down and schedules another consumer. Integer 10
    glide.db.archiving.max_consumer_workers The maximum number of concurrent consumers cluster-wide. Integer 4