MID Server support for Data Stream actions

  • Release version: Yokohama
  • Updated January 30, 2025
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of MID Server support for Data Stream actions

    ServiceNow customers can use a MID Server to retrieve data when running a Data Stream action. The MID Server fetches data and sends each page as an attachment to the instance for processing. When using Data Stream actions within a flow, actions inside a For Each block always execute on the instance, while certain steps like Action Preprocessing Script, Request Script, and REST or SOAP steps can run on either the MID Server or the instance.

    Show full answer Show less

    It is important to minimize switching execution back and forth between the MID Server and instance during data processing, as this can negatively impact performance.

    Key Considerations

    • Execution Environment: Choose either the MID Server or instance for related steps to avoid frequent environment switching.
    • Size Limits and Timeouts: Adjust system properties to handle large data pages or slow connections:
      • Increase the REST transaction quota rule timeout (default 60s, max 300s) to avoid REST step timeouts.
      • Increase the instance timeout for retrieving data pages from the MID Server by modifying datastreamalternativeenvfetchpagetimeoutseconds (default 600s, max 7200s).
      • Increase the maximum allowed attachment size with the com.glide.attachment.maxsize property (default 1024 MB) to accommodate large data pages.
    • JDBC Operations: For JDBC data retrieval, run the Data Stream action asynchronously and poll the Attachments table for results. Timeout properties can be adjusted:
      • com.snc.processflow.datastream.payload.timeout.seconds: Wait time for payload availability (default 600s, max 7200s).
      • com.snc.processflow.datastream.asyncchild.timeout.seconds: Time allocated for child plan execution on the MID Server (default 60s, max 7200s).

    Practical Benefits

    This capability enables customers to efficiently retrieve large or complex data sets through a MID Server while maintaining control over execution environment and performance. By tuning timeout and size properties, customers can avoid common issues such as slow data transfer, timeouts, and attachment size limits, ensuring reliable and scalable Data Stream action execution.

    Get data through a ServiceNow® MID Server when running a Data Stream action.

    When selected, the MID Server retrieves the data and sends each page to the instance as an attachment to process.

    Important:
    When running the Data Stream action within flow, the actions in the For Each block always run on the instance.

    These steps can run on either the MID Server or the instance:

    • Action Preprocessing Script
    • Request Script step
    • REST or SOAP step
    Note:
    Avoid shifting the execution environment between the instance and the MID Server multiple times. For example, you might configure the Request Script step to run on the MID Server, but the REST step to run on the instance. In this case, the system shifts environments between the instance and MID Server for every page of data, which may degrade performance.

    To learn more about running a step on a MID Server, see Integration steps.

    Size limits

    If your system encounters a timeout or size limit issue, try making one of these adjustments:

    Increase the REST transaction quota rule

    If the connection between the instance and the MID Server slows down, you may encounter a timeout while waiting for a response from the REST step. Increase the default timeout by updating the REST Attachment API request timeout transaction quota rule.

    • Default: 60 seconds
    • Maximum: 300 seconds
    Increase the instance timeout

    By default, the instance waits for 600 seconds to retrieve a single page of data from a MID Server. If you encounter a timeout when running a Data Stream action through a MID Server, change this default by increasing the datastream_alternative_env_fetch_page_timeout_seconds system property.

    • Default: 600 seconds
    • Maximum: 7200 seconds
    Increase attachment size
    The MID Server sends each page of data to the instance as an attachment. If a page of data is large, it may exceed the allowed attachment size. Increase the allowed size by updating the com.glide.attachment.max_size system property.
    • Default: 1024 MB
    • Maximum: None

    JDBC operations and MID Server timeouts

    For JDBC operations, execute the Data Stream Action asynchronously and poll the Attachments [sys_attachment] table for results.

    The MID Server processes the SQL statement, while the instance/main thread waits for context payloads to be inserted into the attachment table to query the next record.

    You can adjust timeout values for JDBC operations with the following properties.

    com.snc.process_flow.datastream.payload.timeout.seconds
    The amount of time the instance waits for the payload to be available from the JDBC execution in the MID Server. A bounded property with a minimum value of 0 seconds and a maximum value of 7200 seconds. The default time is 600 seconds.
    com.snc.process_flow.datastream.async_child.timeout.seconds
    The amount of time allocated for the execution of a child plan in the MID Server.A bounded property with a minimum value of 0 seconds and a maximum value of 7200 seconds. The default time is 60 seconds.