Custom (Load by Script) type data source
Summarize
Summary of Custom (Load by Script) Type Data Source
The Custom (Load by Script) data source allows users to utilize custom scripts to import various types of data into the import set table. By selecting this data source type, customers can directly write JavaScript in the Data Loader field to fetch data from sources like REST APIs or file attachments.
Show less
Key Features
- Data Loader Script: This script utilizes parameters such as
importsettable,datasource,importlog,lastsuccessimporttime, andpartitioninfoto manage data insertion into the import set table. - Parallel Loading: Customers can enable parallel loading to process data in smaller sections concurrently, enhancing performance and reducing the impact on other tasks. This requires configuration of the import partition method to 'Custom size'.
- Job Management: Users can introduce delays for scheduled jobs and monitor their statuses through specific tables, ensuring effective management of data imports.
- Script Auto-Complete: The script editor supports auto-completion, providing suggestions as you type to streamline the scripting process.
Key Outcomes
By using the Custom (Load by Script) data source, ServiceNow customers can efficiently load and manage large volumes of data with the flexibility of scripting. This capability leads to quicker integration results and minimizes disruption to system performance. Monitoring and managing parallel jobs enhances operational efficiency and provides better control over data import processes.
Use a custom script to store any type of incoming data in the import set table.
When you select a Type of Custom (Load by Script) for your data source, the Data Loader field appears. The Data Loader field is a JavaScript field where you can enter a script to fetch data and insert it into the import set table using the import_set_table input parameter. For example, you can enter a script to get data from a REST API or a file attachment.
Data Loader script
import_set_table: The import set table referred to in the Data Source record. The data is inserted into this table.data_source: The data source referred to in the Data Source record.import_log: The log that records information about data import activity.last_success_import_time: The last time this data source was run successfully.partition_info: The partitioning information for the data. Used for parallel loading.
addColumn(label, maxLength): Adds a string-type column to the import set table.addJSONColumn(label, maxLength): Adds a JSON-type column to the import set table.addXMLColumn(label, maxLength): Adds an XML-type column to the import set table.insert(rowData): Inserts a map (key = column name, value = column value) in the import set table.getMaximumRows(): Returns 20 when you select Test load 20 records. In every other case, returns -1.
Data Loader example
Parallel loading script
parallel_job_loader: The Parallel Jobs [sys_parallel_job] table.data_source: The data source referred to in the Data Source record.import_log: The log that records information about data import activity.last_success_import_time: The last time this data source was run successfully.
The parallel_job_loader parameter has the add(partitionInfo) method, which adds the partition information to the Parallel Jobs [sys_parallel_job] table.
Parallel loading example
In this example, the parallel loading script partitions the data into smaller sections to be loaded in parallel. The Data Loader script loads the data into the import set table and adds a new partition to process data after a 120-second delay.
Parallel loading delays, status, and properties
For most parallel jobs, data is available when the Data Loader script runs, so you don’t need to add any parallel jobs to the Data Loader script.
In cases where the data might not be immediately available, you can create an additional parallel job with a delay in the Data Loader script. When the Data Loader script runs, the parallel job is scheduled to run after the amount of time specified by the delay.
When adding parallel jobs to the Data Loader script, avoid repetitive loops. You can avoid repetitive loops by specifying a constraint. In the previous example, the parallel job in the Data Loader script uses the constraint
(partition.start < 1000). Delays are measured in seconds.
You can check the status of your current parallel import tasks by going to the Concurrent Import Sets [sys_concurrent_import_set] table, selecting your job, and viewing the Parallel Jobs tab.
Delayed jobs may show as Pending depending on the delay’s length of time. Parallel jobs are processed from the event queue, so you can also view your jobs by going to the Events [sysevent] table and filtering by [Queue] [is] [impex_parallel_job_queue].
If a job gets stuck in a Running state for more than 600 minutes, it's automatically put back into a Pending state. You can modify the amount of time a parallel job is allowed to be stuck in a running state with the com.glide.system_parallel_job_inactive_time_minutes property.
If a parallel job can't complete after two tries, the job is set to Error. You can modify the number of tries before having an error state with the com.glide.system_parallel_job_max_retry_count property. The com.glide.system_parallel_job_auto_retry property can be set to false to disable auto retry.
Script auto-complete
As you type the script, the script auto-complete displays the available options, as shown in this example.