Data Stream action design considerations

  • Release version: Xanadu
  • Updated August 1, 2024
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Data Stream action design considerations

    This guidance outlines important considerations for ServiceNow customers when creating and using Data Stream actions to ensure efficient performance and error-free operation. Data Stream actions require a continuous connection to the response stream and have specific constraints related to pagination, scripting, flow integration, and MID Server usage.

    Show full answer Show less

    Considerations when creating a Data Stream action

    • Pagination logic: Write pagination scripts based on third-party API requirements. Use the getNextPage variable (Boolean type) to control when pagination stops, ensuring you avoid infinite loops by setting a clear end condition.
    • Variable access and data types: Access variables using bracket or dot notation (e.g., variables['getNextPage']). Pagination variables are strings by default; convert them to integers for math operations and back to strings afterward.
    • Preprocessing script limitations: Do not include scripts that pause the action or call a MID Server, as this interrupts the required constant connection and causes errors.
    • Page size management: Aim to keep each page of returned data under 1 GB to prevent size limit issues, especially when retrieving data via MID Server. Larger pages may require increasing attachment size limits.
    • Retry policy constraints: Only fixed interval retry strategies are supported, with a maximum total delay of 30 seconds. During retries, Data Stream steps do not enter a waiting state.
    • Configuration caution: Clearing options on the configuration page deletes the associated steps and data in the Data Stream outline. Nesting Data Stream actions is not supported.
    • Testing: Always thoroughly test Data Stream actions before deploying them in production environments.

    Considerations when adding a Data Stream action to a flow

    • Avoid actions with wait conditions inside Data Stream For Each blocks: Actions like Ask for Approval or Wait for Condition pause the flow and are incompatible with the continuous streaming requirements of Data Stream actions.
    • Be cautious with actions running on MID Server: Some steps, such as JDBC, PowerShell, and SSH, always run on a MID Server and cannot be used inside Data Stream For Each blocks. Others, like REST or Script steps, may run on MID Server depending on configuration—verify runtime behavior.
    • Manage performance with large record sets: When iterating over many records in a For Each block, disable flow reporting by setting the com.snc.processflow.reporting.level system property to avoid resource strain and maintain performance.

    Follow these considerations when creating a Data Stream action and when adding a Data Stream action to a flow to prevent errors and performance issues.

    Considerations when creating a Data Stream action

    Create Data Stream actions with these considerations in mind.

    Write pagination logic according to third-party requirements

    Evaluate and understand the format required by your third-party endpoint. For example, you may need to write a script that sets the built-in getNextPage variable to true as long as there is a nextPage token in the response. If the response does not contain the token, then set the variable to false. Access variables in script using bracket or dot notation. For example, variables['getNextPage']. This variable only accepts the Boolean data type. The default value is false.

    Ensure that preprocessing scripts do not call a MID Server or pause the action.
    Data Stream actions require a constant connection to the response stream. They do not support preprocessing scripts that pause the action to call a MID Server or pause the action to wait for a condition or duration. A preprocessing script that pauses the action for any reason returns an error message.
    Convert pagination variable data types to perform math operations

    Pagination variables only support the string data type. To perform math operations, you must convert the value to an integer, perform any required operations, then convert it back to a string.

    variables['offset'] = (parseInt(variables['offset']) + parseInt(variables['limit'])).toString();
    Ensure that the pagination script has an end condition
    Avoid infinite loops in pagination requests by creating a condition that sets the getNextPage variable to false. Cancel any long-running flows. Always test Data Stream actions before using them in production.
    Clear configuration page options carefully
    Clearing an option in a configuration page removes the step from the Data Stream outline and deletes all data associated with the step.
    Consider the size of each page returned
    When configuring pagination options, try to ensure that the size of each page returned is under one GB. This ensures that flow designers who use the action won't encounter size limit issues when retrieving data from a MID Server. If the action returns more than one GB of data per page, the flow designer may need to increase the attachment size limit. For more information, see MID Server support for Data Stream actions.
    Note:
    • Nesting Data Stream actions is not supported.
    • When using the retry policy, ensure that you are aware of these considerations:
      • Only a fixed time interval strategy is honoured for the data stream actions.
      • The total delay (count multiplied by the interval) can't exceed 30 seconds. For example, if you want retry for a maximum of 3 times, the maximum delay for each retry is 10 seconds.
      • Data stream step doesn’t go into the waiting state during retry.

    Considerations when adding a Data Stream action to a flow

    Add a Data Stream action to a flow with these considerations in mind.

    Do not add actions with wait conditions to a Data Stream For Each logic block

    You cannot add an action that pauses the flow to a Data Stream For Each flow logic block. Data Stream actions require a constant connection to the response stream. For example, you cannot use the Ask for Approval or Wait for Condition actions within Data Stream For Each flow logic.

    Some actions and steps always pause a flow and cannot be added to Data Stream For Each flow logic blocks. However, some actions and steps may pause the flow depending on how they are configured. Check the action configuration to ensure it does not pause the flow at runtime.

    Always pauses the flow (unsupported with Data Stream actions) Ask for Approval action
    Ask for Approval step
    Wait for Condition action
    Wait for Condition step
    May pause the flow depending on configuration Create Catalog Task action
    Create Task action
    Create Task step
    Do not add actions that run on the MID Server to a Data Stream For Each logic block

    Some actions and steps always run on a MID Server and cannot be added to Data Stream For Each flow logic blocks. However, some actions and steps may run on a MID Server depending on how they are configured. Check the action configuration to ensure it does not run on the MID Server at runtime.

    Always runs on a MID Server (unsupported with Data Stream actions) JDBC step
    PowerShell step
    SSH step
    May run on a MID Server depending on configuration REST step
    Script step
    SOAP step
    Turn off reporting when consuming a large number of records in the For Each logic block
    Iterating over a large number of records can be resource intensive, especially when the For Each logic block includes complex actions for each iteration. To avoid performance issues, turn off reporting using the com.snc.process_flow.reporting.level system property. For more information, see Workflow Studio flow system properties.