Custom (Load by Script) type data source

  • Release version: Yokohama
  • Updated January 30, 2025
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Custom (Load by Script) type data source

    The Custom (Load by Script) data source type in ServiceNow allows you to use a JavaScript script to fetch and insert any kind of data into an import set table. This method is highly flexible, enabling you to pull data from sources such as REST APIs or file attachments directly into ServiceNow import sets.

    Show full answer Show less

    Key Features

    • Data Loader Script: A JavaScript field where you write the script to load data. The script uses the importsettable input parameter to insert data rows.
    • Input Parameters for Data Loader Script:
      • importsettable: Interface to manipulate the import set table (add columns, insert rows).
      • datasource: Reference to the data source record.
      • importlog: Logs import activities.
      • lastsuccessimporttime: Timestamp of the last successful import.
      • partitioninfo: Used for data partitioning in parallel loads.
    • Import Set Table Methods: Methods to add string, JSON, or XML columns and insert rows, supporting flexible data structures.
    • Parallel Loading: Supports partitioning data into smaller chunks and loading them concurrently to reduce integration time and system impact.
    • Parallel Loading Input Parameters and Methods: Includes paralleljobloader to add partition info to the Parallel Jobs table, improving management of concurrent data loads.
    • Control Over Parallel Job Behavior: Configurable properties allow you to set timeouts for stuck jobs, retry counts, and enable or disable auto retries to maintain import reliability.
    • Status Monitoring: You can monitor the status of parallel import jobs via the Concurrent Import Sets interface under System Import Sets.
    • Script Auto-Complete: Enhances scripting efficiency by providing suggestions and available options as you write the data loader script.

    Practical Benefits for ServiceNow Customers

    • Enables integration of complex or custom data sources not supported by out-of-the-box import types.
    • Improves performance and minimizes system impact by using parallel loading for large data sets.
    • Provides granular control and monitoring of data import processes to ensure reliability and timely error handling.
    • Supports dynamic and flexible data models via script-controlled column definitions and row insertions.
    • Facilitates automation and customization of import processes, improving operational efficiency.

    Use a custom script to store any type of incoming data in the import set table.

    When you select a Type of Custom (Load by Script) for your data source, the Data Loader field appears. The Data Loader field is a JavaScript field where you can enter a script to fetch data and insert it into the import set table using the import_set_table input parameter. For example, you can enter a script to get data from a REST API or a file attachment.

    The Data Loader script loads the data. You can also Enable parallel loading to use a script to partition the data into smaller sections, then load them in parallel. Parallel loading can enable your integrations to finish in less time and create less of an impact on other tasks.
    Note:
    With parallel loading, you must be running a concurrent import and the Partition Method for the import must be set to Custom size. For more information see, Schedule a data import.

    Data Loader script

    The data loader script has the following input parameters.
    • import_set_table: The import set table referred to in the Data Source record. The data is inserted into this table.
    • data_source: The data source referred to in the Data Source record.
    • import_log: The log that records information about data import activity.
    • last_success_import_time: The last time this data source was run successfully.
    • partition_info: The partitioning information for the data. Used for parallel loading.
    The import_set_table parameter has the following methods:
    • addColumn(label, maxLength): Adds a string-type column to the import set table.
    • addJSONColumn(label, maxLength): Adds a JSON-type column to the import set table.
    • addXMLColumn(label, maxLength): Adds an XML-type column to the import set table.
    • insert(rowData): Inserts a map (key = column name, value = column value) in the import set table.
    • getMaximumRows(): Returns 20 when you select Test load 20 records. In every other case, returns -1.

    Data Loader example

    Data Loader example showing a script that inserts data into the import set table.

    Parallel loading script

    The parallel loading script has the following input parameters.
    • parallel_job_loader: The Parallel Jobs [sys_parallel_job] table.
    • data_source: The data source referred to in the Data Source record.
    • import_log: The log that records information about data import activity.

    The parallel_job_loader parameter has the add(partitionInfo) method, which adds the partition information to the Parallel Jobs [sys_parallel_job] table.

    Parallel loading example

    Parallel loading script example.

    Parallel loading properties and status

    If a job gets stuck in a Running state for more than 600 minutes, it's automatically put back into a Pending state. You can modify the amount of time a parallel job is allowed to be stuck in a running state with the com.glide.system_parallel_job_inactive_time_minutes property.

    If a parallel job can't complete after two tries, the job is set to Error. You can modify the number of tries before having an error state with the com.glide.system_parallel_job_max_retry_count property. The com.glide.system_parallel_job_auto_retry property can be set to false to disable auto retry.

    You can check the status of your current parallel import tasks by navigating to All > System Import Sets > Advanced > Concurrent Import Sets, selecting your job, and viewing the Parallel Jobs tab.

    Script auto-complete

    As you type the script, the script auto-complete displays the available options, as shown in this example.

    Data Loader auto-complete