None batch import

Select the none batch import method if you do not want to use the pagination method provided in Automation Center. There is no step required here.

Offset-based batch import

Select the offset-based batch import method (pagination method) if you want to import data using a specific start point (offset) and a batch size (limit).

• Offset: The starting point for importing data.
• Total items: The total number of records that you want to import.
• Limit: The number of records to import in each batch.

For example: If you set the offset to 20, total items to 200, and limit to 10, then the system will import data in batches of 10 records starting from record 20 and continuing until all 200 records are imported.

Steps to configure

1. In the SourceHub get offset based pagination chunks action, specify values for the offset, total items, and limit fields, and then select Done.

   After the values are defined, the chunks_child0 field will be available in the right-hand panel (under For Each).

2. For the action you selected during the Select Action step, map the limit from SourceHub get offset based pagination chunks action and offset values using the chunks_child0 field output generated by the SourceHub get offset based pagination chunks action.

   

   Handling actions that don’t support limit and offset

   Map the limit and offset values in your action using the output returned by the SourceHub get offset based pagination chunks action.

   If your selected action doesn’t have predefined parameters for offset-based pagination but includes a filter that can accommodate these parameters, you can do one of the following:

   • Option 1: Add an action just before your selected action to handle the limit and offset. This action should generate a query that the primary action can use to fetch data.
   • Option 2: Copy your primary action, modify it to include inputs for limit and offset, and process these inputs to create a query. Pass this query to the filter of your original action.
   Note:

   If you choose Option 2, ensure that you create a copy of the original action first. After copying, reselect the copied action in the Select Action step and regenerate the flow. For information about selecting an action, see Select an action.
3. Select Done and Test.
4. Follow steps 3 onwards in Configure batch import for data.

Time-based batch import

Select the time-based batch import method if you want to import data for a specific date range.

Steps to configure

1. In the SourceHub get date based pagination chunks action, specify the start date and end date, then click Done.

   After you values are defined, the range field in your right panel is available.

2. For the action you selected during the Select Action step, map the start date and end date values using the range field output generated by the SourceHub get date based pagination chunks action.

   Date format

   The Connection Manager uses and returns dates in the format: YYYY:MM:DD:HH:MM:SS. If your Action supports date ranges, the format is automatically applied.

   Handling actions that don’t support date ranges

   If your action doesn’t automatically support date ranges but includes a filter for custom parameters, you can follow one of these approaches.

   • Option 1: Add an action before your selected action to handle the start date and end date. This action should generate a query that your primary action can use to fetch data.
   • Option 2: Copy your primary action and modify it to include inputs for start date and end date. Process these inputs within the action to create a query and pass it to the filter of your original action.

     

     Note:

     If you choose Option 2, ensure that you create a copy of the original action first. After copying, reselect the copied action in the Select Action step and regenerate the flow. For information about selecting an action, see Select an action.
3. Select Done and Test.
4. Follow steps 3 onwards in Configure batch import for data.

Token-based batch import

Select the token-based batch import method if you want to import data using a continuation token.

When you select the token-based batch import method on the Configure batch import page, you must provide a name for the token. This token represents the continuation value returned by your action (such as, next_url, next_token, or next). Ensure that the token name matches exactly as it is returned from the action, including spelling and capitalization.

Steps to configure

1. If the provided token name is correct, the do-while condition will automatically map it. If it remains blank, manually drag the continuation token from your action’s output into the condition field of the do-while logic.
2. Add a Flow Variable of type string to store the updated value of the token.
3. Save your changes, and click Done.
4. Insert a Set Flow Variable logic step after your selected action. In this step, set the value of the newly created flow variable to the continuation token returned by your action. Ensure this token matches the one mapped to the do-while condition.

   

5. Select Done and Test.
6. Follow steps 3 onwards in Configure batch import for data.