How to catch up with missing data using Bulk Load Flow
To restart flow that is behind for more than 1-2 days it requires to restart it from current date and use bulk load utility flow to catch up with previous data. Restarting scheduled flows from the current date/time is to avoid integration server overload with back log of data that CANNOT be transferred by scheduled flow.
You need to run particular DR (Date Range) or DRS (Date Range + batch Size) bulk load utility flow (with DR or DRS at the end of its name) to catch up with missing data. Bulk load flows are designed to move big amount of data without integration server overload. I have attached the instruction on how to run bulk load flow.
In order to bulk load only the Items of the specific type the prefix to the Start Date parameter must be used. Here is the list of prefixes:
- I – inventory items only
- N – non-inventory items only
- S – service items only
- G – group items only
- L – all item types except group items
- R – all item types except inventory items
Examples for Start Date Field (as for most of the cases, if Date includes time, then modified timestamp is used; otherwise the created one):
- I2023-01-01 – load all Inventory items that were created on or after 2023-01-01
- S2023-01-01 09:00:00 – load all service items that were last modified on or after 2023-01-01 09:00:00
Catch-up Procedure Steps:
- Stop scheduled flows and change their Query Starts date to current date if it is older than 1-2 days
- Use DR/DRS bulk load flow to catch up with previous data
- Run one bulk load flow at a time
- Restart scheduled flows after bulk loading is done
