Handle pagination when reading from an API in Jitterbit Studio

Introduction

Most REST APIs cap the number of records returned in a single response. When a dataset is larger than that limit, the API splits the results across multiple pages and expects the caller to request each page in sequence. Without pagination support, a Studio operation retrieves only the first page and silently drops the rest.

This guide covers the page-number pagination approach, where each request includes a page parameter that increments with each call. A note at the end of Part 2 describes how to adapt the pattern for cursor-based APIs.

This guide assumes familiarity with configuring HTTP v2 connections and activities. For a general introduction, see Call a REST API using the HTTP v2 connector.

Design pattern

The pattern uses two operations:

Fetch page operation: Reads one page from the API and writes the records to the target, using the transformation pattern:

flowchart LR A[HTTP v2 GET activity] --> B[Transformation] --> C[Target activity]
Controller operation: A single script step that loops until all pages are retrieved.

The controller script uses a While loop that calls RunOperation on the fetch page operation for each page. RunOperation runs synchronously by default, so global variable changes made inside the fetch page operation (including the signal that no more pages remain) are visible back in the controller after each call.

Two global variables coordinate the loop:

page: the current page number, initialized to 1 in the controller and incremented after each fetch.
has_more: a flag initialized to true and set to false by the transformation when the last page is detected.

Part 1: Configure the fetch page operation

Step 1: Configure the HTTP v2 GET activity

On the design canvas, drag an HTTP v2 GET activity from an existing endpoint onto the canvas to start the fetch page operation.
Double-click the activity to open its configuration.
Name: Enter a name such as Get Contacts Page.
Path: Enter the API endpoint path, for example /contacts.
Request Parameters: Click the add icon to add a row, and enter the following:
- Name: page
- Value: $page
This passes the global variable page as a query parameter on each request. Add a second row with Name per_page and a fixed value such as 100 to control the number of records returned per page.
Click Next.

Step 2: Define the response schema

Select Yes, Provide New Schema.

Enter a JSON schema that includes the records array and the pagination metadata field returned by the API. The following example schema represents a response that includes a contacts array and a total_pages field:

{
  "type": "object",
  "properties": {
    "contacts": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": { "type": "string" },
          "name": { "type": "string" },
          "email": { "type": "string" }
        }
      }
    },
    "total_pages": { "type": "integer" }
  }
}

Adjust the schema to match your API's actual response structure.

Click Next, then click Finished.

Part 2: Extract pagination state in the transformation

The transformation maps the source fields from the GET activity to the target activity and uses a script to update the has_more global variable based on whether more pages remain.

Open the transformation that follows the GET activity.
Map each field from the source contacts array to the corresponding target fields.
In the transformation, add a script to the root level (outside the looping node) that sets has_more based on the total_pages value from the response:
```
$total_pages = Source.total_pages;
If($page >= $total_pages, $has_more = false);
```
This reads the total_pages value from the response and sets $has_more = false when the current page is the last.
Tip

If the API does not return a total page count but simply returns fewer records than the page size when the last page is reached, use the record count to detect the end instead:
```
If(Count(Source.contacts.id) < 100, $has_more = false);
```
Replace 100 with the per_page value configured in the GET activity.

Cursor-based pagination

Some APIs return a cursor or a next_page URL in the response rather than a total page count. To adapt this pattern for cursor-based APIs, replace page with a cursor global variable (initialized to ""), pass cursor as a request parameter named cursor, and in the transformation map the response's next_cursor field to cursor. Set has_more = false when cursor is empty:

$cursor = Source.next_cursor;
If($cursor == "", $has_more = false);

Remove the $page++ increment from the controller script described in Part 3.

Part 3: Write the controller script

The controller operation contains a single script that initializes the loop state and calls the fetch page operation repeatedly until all pages have been processed.

On the design canvas, create a new operation containing only a Script step.

Double-click the script to open the editor and enter the following:

$page = 1;
$has_more = true;

While($has_more,
    If(!RunOperation("<TAG>operation:Fetch Page</TAG>"), RaiseError(GetLastError()));
    $page++;
);

Replace Fetch Page with the exact name of the fetch page operation.

Save the script.

Key points about this script:

page and has_more are global variables inherited by the fetch page operation on each synchronous RunOperation call.
After each page is processed, the controller increments page before starting the next iteration.
RaiseError(GetLastError()) stops the loop immediately and surfaces the failure if the fetch page operation returns an error.

The While function enforces a maximum iteration count, which defaults to 50,000. For APIs with very large datasets, set $jitterbit.scripting.while.max_iterations to an appropriate limit before the loop:

$jitterbit.scripting.while.max_iterations = 2000;
$page = 1;
$has_more = true;

While($has_more,
    If(!RunOperation("<TAG>operation:Fetch Page</TAG>"), RaiseError(GetLastError()));
    $page++;
);

Verify the integration

Deploy and run the controller operation. Check the operation logs: the fetch page operation appears once per page, so the log entry count confirms how many pages were retrieved. Verify that the total record count in the target matches the full dataset expected from the API.

To test early termination before processing a full dataset, set $jitterbit.scripting.while.max_iterations = 3 in the controller script on the first run. This limits the loop to three pages and lets you confirm that the page increment, schema mapping, and has_more logic are all working correctly before removing the limit and running against the full dataset.

To add automatic retries if a page fetch fails, see Retry a failed operation. For guidance on constructing query strings with dynamic cursor or filter values, see Build dynamic query strings for REST API calls.