Archiving records

Release version: Xanadu

Updated August 1, 2024

4 minutes to read

Summarize

Summarized using AI

Summary of Archiving records

Archiving records in ServiceNow helps manage table size growth and improves query performance by moving older or less frequently accessed data to archive tables. This feature supports archiving records in core tables likeTask [task]and custom tables created on the ServiceNow AI Platform using archive rules. Specialized tools exist for archiving Configuration Management Database (CMDB) CI records and emails, which require different processes.

Show full answer Show less

How Archiving Works

Activating an archive rule creates a new archive table prefixed with "ar" (e.g., arincident for the Incident table).
An XML version of each archived record is stored in the sysarchivelog table, which uniquely stores the sysid and display values for reference fields.
Multiple joined tables are flattened into a single archive table, eliminating base and extended table structures.
Reference field values are converted to their display strings at the time of archiving, ensuring that archived data reflects the state of references (e.g., user names) when archived, not current values.
A new module is added to the System Archiving application for easy access to each archive table's list and form views.

Handling Reference Fields in Archives

Archived data stores reference fields as strings representing the display value at archive time. For example, a caller in an incident appears as the user name when archived, even if the actual user record changes or is deleted later. This approach preserves historical accuracy but means archived data does not reflect updates to referenced records.

Querying Archived Data

Archive tables are not optimized for broad ad hoc queries. They only index display values, creation dates, and sysid. Customers should avoid complex queries (e.g., searching all priority 1 archived incidents) and instead search using indexed fields like specific record numbers or creation dates for efficient performance.

Access Control on Archive Tables

By default, archive tables inherit Access Control Lists (ACLs) from their original tables.
To explicitly manage access to archive tables, enable the glide.security.enablearchivetableacls property and define archive-specific ACLs.
If archive table ACLs exist, they take precedence; if none are defined, the system uses the original table’s ACLs.
Enabling this feature on upgraded instances requires manual property configuration; new (Zbooted) instances have it enabled by default.

Setting Language for Archived Strings

On internationalized instances, archive display values use the SYSTEM user’s language setting or default instance language if SYSTEM user is not present. Customers can configure language preferences by setting a SYSTEM user with the desired language or adjusting the system default language.

Managing Archive Rules and Data

Create archive rules to define criteria for which records to archive.
Archive rules can also archive, clear, or delete related records.
Customers can adjust archive schedules, pause rules, or restore archived data as needed.
For preserving data integrity, especially for reference fields, there is an option to manually migrate non-reference fields to reference fields by preserving sysid instead of display names.

Practical Benefits for ServiceNow Customers

Improves system performance by reducing active table size.
Maintains historical accuracy of archived data by converting references to fixed display values.
Provides control over data lifecycle with configurable archive rules and retention options.
Ensures security and compliance by allowing customized ACLs for archived data.

Manage table size growth and improve query performance by archiving records.

You can archive records in core tables such as the Task [task] table and records in custom tables that you create on the ServiceNow AI Platform using archive rules.

To archive Configuration Management Database (CMDB) CI records, use the CMDB Data Manager. See Working with CMDB Data Manager.
To archive emails, activate the Email retention plugin and use the archive and destruction rules that come with the plugin. Don’t use the archive feature to create your own archiving rules on the email table.

Archive activation

When you activate an archive rule, the system performs the following actions:

Creates the archive table in the database. The archive table has the same name as the primary table with an "ar_" prefix. For example, if you archive the Incident [incident] table, the archive table is [ar_incident].
Stores an XML version of each archived record in the sys_archive_log table. This archive log is the same table for all archive rules, and you can’t alter this behavior. It’s also the only place where the sys_id is stored together with the display value for reference fields.
For example, for ar_incident <assigned_to>Fred Luddy</assigned_to>, the sys_archive_log record is as follows:
```
<assigned_to display_value="Fred Luddy">5137153cc611227c000bbd1bd8cd2005</assigned_to>
```
Converts multiple joined tables into a single flat-file archive table. The archive table no longer consists of a base table and extended tables.

Figure 1. Conversion of Multiple Joined Tables into a Flat Archive Table
Converts reference field values (values set by references to records in other tables) into string values. The archive record contains the display value of the reference field at the time of the archive.
Adds a module to the Archive Tables list in the System Archiving application. The module name is a combination of the word "Archive" plus the display name for the archived table. For example, the archive module for the Attachment [sys_attachments] table is Archive Attachment.
Creates a list of the archive table using the default list view.
Creates a form for the archive table using the default form view. The form excludes any dot-walking fields such as Caller ID.Email.

Reference values converted to strings

Archived data is stored as a flat file with no reference fields to other tables. The archive process converts any references to other tables to string values.

In the case of a reference field, the string uses the display value such as the caller's user name. For example, the Caller reference field in an incident would display the string ITIL User. If the reference was a document ID and the archive rule included the option to archive related document IDs, the string is the document ID of the related record.

Future changes to reference values aren't reflected in archived records. For example, if you change the user name for "John Smith" to "John A Smith", all active incident records automatically show the caller as "John A Smith" because of the reference between the Incident and User tables. However, all archived incident records display the user name that existed at the time of the archive. Likewise, if you delete a user from the system, current incidents no longer display the deleted user as a caller. However, archived incidents still display the string "John Smith" because that value was used when the record was archived.

Querying archived data

Archived tables aren’t optimized for ad hoc queries. They only contain index entries for the display value, creation date, and the primary key of sys_id.

Avoid making on-demand queries against an archived table, like searching for all priority 1 archived incidents. Instead, only search against the indexed fields. For example, search for incident INC100001 or incidents created on a specific date.

Archive tables and ACLs

By default, archive tables use the ACLs for the unarchived table of the same name. For example, the archived Incident [ar_incident] table uses the ACLs defined for the unarchived Incident [incident] table.

You can manage access to archive tables explicitly by creating ACLs for specific archive tables and setting the glide.security.enable_archive_table_acls property to true. The system then follows one of two paths:

If one or more active ACLs are defined for an archive table, those ACLs control access to the archive table.
If no ACLs are defined for an archive table, the system reverts to default behavior and uses the ACLs for the unarchived version of the table.

Note:

The two paths are mutually exclusive: If archive table ACLs deny access, the system doesn't attempt to revert to the default behavior.

The read operation is the only operation evaluated, and other operations are prevented.

The Execution Plan UI is aware of this logic and presents information accordingly. For example, adding the first ACL to an archive table shows that the archive table ACL is "masking" ACLs on the unarchived (original data) table.

If you have existing ACLs on archived tables, they’re ignored unless you set the glide.security.enable_archive_table_acls property to true. Those newly activated ACLs may possibly cause access issues. To prevent this occurrence, the system sets the glide.security.enable_archive_table_acls property as follows:

Instances without the glide.security.enable_archive_table_acls property use the default value of false.
Upgraded instances don't install the property. The property must be added manually and set to true to enable the archive table ACL behavior.
Zbooted instances install the property and set it to true.

Setting the language of archived strings

On internationalized instances, the archive process uses the language of the SYSTEM user to select the display value strings.

If there’s no SYSTEM user, the instance uses the default language setting to select the display value strings. You can either create a SYSTEM user with a specific language setting or set the system default language to select the language of archived strings.