Parallel exports
Summarize
Summary of Parallel exports
Parallel exports allow you to split outgoing data into multiple export sets and process them simultaneously, significantly reducing processing time. This feature is particularly beneficial for large datasets (50,000 or more records) that take longer to export due to complex scripts. If the dataset is below this threshold, the system defaults to standard export processing.
Show less
Key Features
- Concurrent Processing: Data is divided into chunks based on your instance node configuration, calculated as the number of nodes multiplied by a scale factor (default is one). A minimum of two chunks is required.
- Export Set Creation: Each parallel export generates a Parallel Export Set record and multiple job records for each chunk, allowing for efficient processing of large datasets.
- Export History Records: For each chunk, an export history record is created, which includes a file attachment of the exported data.
- Customizable Threshold: The default threshold for parallel export is set at 50,000 rows, but it can be adjusted by creating a system property.
Key Outcomes
By utilizing parallel exports, ServiceNow customers can expect:
- Reduced export processing times for large datasets.
- Organized export records that link back to the corresponding jobs and sets.
- Files stored efficiently on the MID Server, with a clear naming convention that includes relevant identifiers and timestamps.
Split outgoing data into multiple export sets and process export sets in parallel to reduce processing time.
Running a parallel export can be helpful when exports take a long time due to large datasets with time-consuming scripts.
Parallel export is most effective for exports with 50,000 or more records. If your export doesn't meet the minimum requirements, the system automatically uses standard export processing.
How parallel exports work
When you enable parallel export, the system splits your data into multiple chunks and processes them concurrently to reduce processing time. The number of chunks depends on your instance node configuration and is calculated as: Number of chunks = (Number of nodes) × (Scale factor). The default scale factor is one, and a minimum of two chunks is required.
- A two-node cluster produces four export sets
- A ten-node cluster produces 10 export sets
- Poll the Parallel Export Sets Jobs queue
- Pick export sets from the queue
- Process the export sets
All jobs run concurrently, depending on the availability of worker threads.
Parallel export record structure
- All related export sets
- Parallel export set jobs
- Export histories
Parallel export requires 50,000 rows by default. This threshold enables parallel processing to be used only for large datasets where it provides significant performance benefits.
To customize this threshold, create the system property glide.scheduled_export.min_rows_for_parallel_export with an integer value.
- A parallel export set record is created with a PESO prefix (example: PESO010001)
- Multiple parallel export-set job records are created, one per chunk, with a PESJ prefix (example: PESJ0010001, PESJ0010002)
- Multiple export history records are created, one per chunk
- Each export history record contains an exported file attachment
The Parallel Export Set and Parallel Export Set Job fields on the export history records link back to the parallel export set and individual jobs.
Files from parallel exports are stored on the MID Server in a parallel subfolder: {MID_Server}/agent/export/parallel/{configured_path}/. File naming format includes the parallel export set number, export set name,
timestamp, and sequential file number: Example: PESO0100001_incident__20251204001638_1.xlsx. The file number increments for each chunk (_1, _2, _3, and so on).