Configure operation chunking for large datasets in Jitterbit Studio

Introduction

By default, Studio processes all source records through a transformation in a single pass before writing to the target. For large datasets, this can exhaust available memory or exceed the per-call record limits imposed by APIs such as Salesforce (200 records per call) and NetSuite.

Chunking breaks the source dataset into smaller batches. Each batch is transformed and written to the target independently. This allows Studio to:

• Process datasets larger than available memory by keeping only one batch in memory at a time.
• Meet API record limits by sending one API call per chunk rather than one call for the entire dataset.
• Improve throughput by processing multiple chunks in parallel using multiple threads.

Chunking is available for operations that contain a transformation or an activity from one of the following connectors: Database, NetSuite, Salesforce, Salesforce Service Cloud, ServiceMax, or SOAP.

For Salesforce, Salesforce Service Cloud, and ServiceMax operations, chunking is enabled automatically with default settings.

Configure chunking

1. On the design canvas, right-click the operation and select Settings to open the operation settings.

2. Select the Options tab.

3. Under Enable Chunking, enable the toggle.

4. Configure the following fields:

   • Chunk Size: The number of source records per batch. Set this to match your API's per-call record limit, or to a size that fits comfortably in available memory. The default is 1 for most connectors and 200 for Salesforce operations.
   • Number of Records per File: The number of records to write to each target file. Leave at 0 (no limit) unless your target connector requires output files to be split at a specific record count.
   • Max Number of Threads: The number of chunks to process simultaneously. The default is 1 (sequential). Increasing this uses multiple CPU cores for parallel processing but has implications for variable behavior (see Variable scoping with chunking below).

5. Click Save.

Choose the right chunk size

The right chunk size depends on your use case:

• API record limits: Set the chunk size to match the API's maximum records per call. For Salesforce standard activities, use 200. For Salesforce bulk activities, use a large value such as 10000. Bulk activities are designed for high-volume loads, and the default chunk size of 200 results in an unnecessarily high number of API calls.
• Memory constraints: Use a chunk size small enough that one chunk's data fits in the agent's available memory. For very wide records (many fields or large text fields), a smaller chunk size reduces peak memory usage.

• Database targets: For database upsert operations, set the Jitterbit variable jitterbit.target.db.commit_chunks to true in a script step before the operation. This commits each chunk to the database as it becomes available rather than accumulating all chunks in a temporary file first, which significantly improves throughput for large loads:

  $jitterbit.target.db.commit_chunks = true;
  

  See Perform a bulk upsert to a database for a full walkthrough.

Use multiple threads for parallel processing

Setting Max Number of Threads above 1 allows multiple chunks to be processed simultaneously. This can improve performance significantly when records are independent of each other and the target system can accept concurrent requests.

Variable scoping with chunking

When chunking is enabled, each chunk runs in its own thread. Global variables and project variables are not shared between threads: each thread receives its own copy of the variables as they existed at the start of the transformation.

When the operation completes, only the variable values from the first thread are preserved. Changes made in any other thread are discarded.

This means:

• Counters or accumulators updated inside the transformation (for example, $record_count++) will only reflect updates from the first chunk when the operation ends.
• If you need to accumulate values across all chunks, do it outside the chunked operation, for example in a controller script that reads the results after the operation completes.

To create variables that are scoped to individual chunks (for example, to generate a unique filename per chunk), use the SCOPE_CHUNK variable prefix or the GetChunkDataElement and SetChunkDataElement functions.

Verify chunking is working

Deploy and run the operation and open the operation logs. When chunking is active, the log shows one entry per chunk processed. The total number of log entries confirms how many chunks the dataset was split into.

Verify that the total record count written to the target matches the full source dataset. If records are missing, check that the chunk size is not set larger than the dataset. In that case, only one chunk processes and the behavior is identical to running without chunking.

For further guidance on operation performance and error handling, see Configure error handling in operations and Retry a failed operation.