Store and retrieve session state using Cloud Datastore in Jitterbit Studio

Introduction

Cloud Datastore is Jitterbit's built-in cloud storage system. It lets Studio operations persist and retrieve data between runs without requiring an external database. This guide covers the common pattern for storing session state (such as conversation history, per-user flags, or cross-session context) using a Cloud Datastore key storage.

Key storage is the appropriate storage type for session state. Data in a key storage is retained until it is explicitly deleted, which means records persist indefinitely across sessions. Status storage, by contrast, is intended for tracking operation statuses and is automatically deleted after 90 days, making it unsuitable for durable session records.

Design pattern

Three operations implement the session state pattern:

The Query operation looks up the session record by a unique key (typically a user ID or session ID) and checks whether a record for that key already exists. A transformation before the Query Items activity sets the query filter, and a script after it reads totalItems from the query response: if the value is 0, the Insert operation runs to create a new record; if the value is greater than 0, the Update operation runs to overwrite the existing record with new session data.

Part 1: Create the key storage

Before configuring the connector in Studio, create a key storage and access token in the Management Console.

1. In the Harmony portal, navigate to Harmony portal menu > Management Console > Cloud Datastore.

2. On the Storages tab, click Add Storage, then select Create Key Storage.

3. Storage name: Enter a name for the storage (for example, ConversationHistory).

4. Environment: Select the environment that will use this storage. This field cannot be changed after saving.

5. Description: Optionally enter a description.

6. Under Fields, click Add Field for each custom field needed to store session data. For conversation history, add a field named ConversationHistory with the type Big Text. The built-in Key, Alternative Key, and Value fields are always present and do not need to be added.

   Note

   A maximum of 20 custom fields are allowed per storage, and each item is limited to 25,000 bytes.

7. Click Save.

8. Navigate to Harmony portal menu > Management Console > Access Tokens and create a new access token scoped to the same environment. Copy the token value.

   Tip

   Store the access token as a project variable with its value hidden, then reference it in the connection configuration. This makes it easy to rotate the token without editing the connection directly.

Part 2: Configure the Cloud Datastore connection

1. In Studio, open your project and click the Project endpoints and connectors tab in the design component palette.

2. Click the Cloud Datastore connector to open the connection configuration.

3. Connection Name: Enter a name for the connection (for example, Cloud Datastore).

4. Access token: Enter the access token generated in Part 1, or reference it using a project variable.

5. Click Test to verify the connection, then click Save Changes.

Part 3: Query for an existing session record

The Query operation determines whether a session record for the current user already exists.

Configure the Query Items activity

1. On the design component palette, expand the Cloud Datastore endpoint you created. Drag the Query Items activity type onto the design canvas.

2. Double-click the activity to open its configuration.

3. Name: Enter a name for the activity (for example, Query Session).

4. Select storage: Select Select existent storage, then click the storage created in Part 1 in the table.

5. Click Next to review the data schemas, then click Finished.

Map the query filter in a transformation

Place a transformation before the Query Items activity to define which record to look up. In the transformation:

• Map fields.item.key to the literal string "Key". This is the built-in key field name in a key storage and is case-sensitive.
• Map fields.item.value to the session identifier (for example, the user ID or channel ID from the inbound request payload).
• Map limit to 1. Only one record per session key is expected.

For example, if the inbound request provides the user ID in a variable userId:

<trans>
$userId
</trans>

Map this script node to fields.item.value in the transformation.

Check the query result in a script

After the Query Items activity, add a Script step to the operation. The script reads totalItems from the query response and routes execution to either the Insert or Update operation:

<trans>
if(Source.json.pagination.totalItems == 0,
  RunOperation("<TAG>Operations/Insert Session</TAG>"),
  RunOperation("<TAG>Operations/Update Session</TAG>")
);
</trans>

Replace Insert Session and Update Session with the actual names you give those operations in Part 4 and Part 5. The <TAG> syntax resolves the operation by name at runtime.

Part 4: Insert a record for a new session

The Insert operation runs when no record exists for the session key.

Configure the Insert Items activity

1. Drag the Insert Items activity type onto a new operation canvas.

2. Double-click the activity to open its configuration.

3. Name: Enter a name for the activity (for example, Insert Session).

4. Select storage: Select the same storage used in Part 3.

5. Click Next to review the data schemas, then click Finished.

Map the session data

In the transformation that precedes the Insert Items activity, map the following request fields:

• Key: Map to the session identifier (for example, userId). This is the value used to look up the record in future queries.
• ConversationHistory (or whichever custom fields your storage defines): Map to the initial session data to store.

Leave AlternativeKey and Value unmapped unless your use case requires them.

Part 5: Update an existing session record

The Update operation runs when a record for the session key already exists.

Configure the Update Items activity

1. Drag the Update Items activity type onto a new operation canvas.

2. Double-click the activity to open its configuration.

3. Name: Enter a name for the activity (for example, Update Session).

4. Select storage: Select the same storage used in Part 3.

5. Click Next to review the data schemas, then click Finished.

Map the updated session data

In the transformation that precedes the Update Items activity, map the following request fields:

• Key: Map to the session identifier. This field identifies which record to update: it must match the value used when the record was inserted.
• ConversationHistory (or whichever custom fields your storage defines): Map to the updated session data. All mapped fields are overwritten.

Verify the integration

1. Deploy and run the Query operation manually, passing a session identifier that does not yet exist in the storage.

2. In the operation log, confirm that totalItems is 0 and that the Insert operation was triggered.

3. In Management Console > Cloud Datastore, open the storage details and confirm that a new record with the expected key and field values appears.

4. Deploy and run the Query operation again with the same session identifier.

5. In the operation log, confirm that totalItems is 1 and that the Update operation was triggered.

6. In Management Console > Cloud Datastore, confirm that the record's custom fields reflect the updated values.

7. If the Query activity fails with an authentication error, verify the access token in the Cloud Datastore connection and confirm the token is associated with the same environment as the storage.

Cloud Datastore can also be used for cross-run deduplication at high volume, as an alternative to cache functions. See Detect and deduplicate records using hash functions.