Batching¶

Introduction¶

Batching, or batch processing, is an optional feature that can process very large sets of data through CRUD rules in the business layer. It can also help prevent deadlocks by freeing the connection to the table or database on both the source and target. Batching is not enabled by default.

How to enable batching¶

To enable batching on a CRUD rule, follow these steps:

1. Navigate to the CRUD rule in the business layer.

2. Select More > Edge Case.

3. In the Edge Case Settings dialog, set values for the following:

   • Source Batch Size: The number of records to be included in a single batch. (Recommended: less than 32000.)

   • Batch mode: Select one of the following options:

     • Consume rows: If the result set changes after the CRUD rule executes. Use this, for example, when you want to update all rows with column 'X' = 3 to have a value of 2 on that same row. After the first batch runs, the result set will change, so you must consume rows instead.

       When set, the following field appears:

       • Batch Limit: Maximum number of batches. Set this to prevent infinite loops. For example, if your rule is expected to modify 100000 rows of data, and Source Batch Size is 5000, setting Batch Limit to 30 limits the number of consumed rows to 150000.

     • Cycle through rows: If the result set does not change after the CRUD rule executes. Use this, for example, when you want to update a column 'X' of every row that has a column 'Y' = 3. That source will not change throughout (running the results before and after will yield the same results), so you can cycle through the rows.