Detect and deduplicate records using hash functions in Jitterbit Studio

Introduction

Hash functions give each record a short, fixed-length fingerprint derived from its field values. Two records with identical field values produce identical fingerprints; a record whose values have changed produces a different fingerprint from the one computed on the previous run. Studio provides two cryptographic hash functions for this purpose: SHA256, which returns a 64-character hex string, and MD5, which returns a 32-character hex string.

This guide covers two patterns built around these functions:

Part 1: Cross-run change detection: Compute a hash of each source record during a sync run, compare it against the hash stored in cache from the previous run, and skip records whose hash has not changed.
Part 2: Within-run deduplication: Track which records have already been processed during a single run using cache, and skip duplicate records encountered later in the same dataset.

Both patterns use the Harmony cloud cache functions ReadCache and WriteCache to store state. Cache calls are subject to a combined limit of 100 calls per minute per organization; for high-volume or long-lived storage needs, Cloud Datastore is the recommended alternative.

This guide assumes familiarity with transformation conditions. For background, see Filter records using conditions.

Design pattern

Both patterns use a single operation:

flowchart LR A[Source activity] --> B["Transformation
(with condition)"] --> C[Target activity]

The transformation condition node runs once per source record. The condition script reads from cache, evaluates whether to process the record, and returns true (process) or false (skip). When a record is processed, the script also writes updated state to cache for use on the next evaluation.

The cache logic lives in the transformation condition rather than in a separate script step. This keeps the state management close to the data and avoids the complexity of coordinating which records were processed across a step boundary.

Part 1: Cross-run hash-based change detection

This pattern detects whether a source record has changed since the last integration run. A hash of each record's fields is stored in cache with a 30-day expiration so that it persists between runs.

Step 1: Choose a hash function

Use SHA256 unless processing speed is the primary concern. SHA256 returns a 64-character hex string and has no known practical collision vulnerabilities for this use case. MD5 returns a 32-character hex string and is slightly faster, but produces a weaker fingerprint.

For most integration volumes, the performance difference is negligible. Use SHA256 by default.

Step 2: Construct the hash input

Concatenate the fields that should trigger processing when they change. Use | as a separator between field values to prevent adjacent values from producing the same concatenated string by accident.

For example, if a contact record should be processed when email, first_name, or last_name changes:

Source.id + "|" + Source.email + "|" + Source.first_name + "|" + Source.last_name

Note

Include every field that should trigger processing when changed. Fields omitted from the hash input are invisible to the change detector: changes to those fields are silently ignored.

Step 3: Write the condition script

Open the transformation and add a condition script to the root-level node. The script reads the stored hash, computes the current hash, and returns true or false.

Replace Source.id, Source.email, Source.first_name, and Source.last_name with the actual field names from your source schema. Replace "my_project" with a stable prefix that identifies the integration: this prevents cache key collisions with other operations in the same environment.

// Build the cache key using a stable prefix and the record's unique ID
cacheKey = "my_project_contact_hash_" + Source.id;

// Read the hash stored from the previous run (env scope, 30-day expiration)
storedHash = ReadCache(cacheKey, 2592000, "env");

// Compute the hash of the fields that matter for this record
currentHash = SHA256(Source.id + "|" + Source.email + "|" + Source.first_name + "|" + Source.last_name);

// Write the current hash back to cache.
// If the record is unchanged, this renews the cache expiration timer.
// If the record changed, this updates the stored hash for the next run.
WriteCache(cacheKey, currentHash, 2592000, "env");

// Return true to process the record, false to skip it
If(storedHash == currentHash, false, true);

Tip

To use MD5 instead of SHA256, replace SHA256(...) with MD5(...). The rest of the script remains identical.

Key points about this script:

ReadCache returns null when no value is stored for the key (for example, on the first run or after a cache expiration). A null stored hash never equals the computed hash, so all records are processed on the first run.
WriteCache is called for every record, including unchanged ones. For unchanged records, this renews the 30-day expiration without modifying the stored value.
2592000 is the maximum cache expiration in seconds (30 days). Adjust this value if your sync runs less frequently and records may not appear in consecutive runs.
The "env" scope makes the cache entry visible to all operations in the same environment and causes it to persist between separate runs. The default "project" scope does not persist between runs.

Warning

ReadCache and WriteCache combined are limited to 100 calls per minute per organization. This pattern makes two cache calls per record: one read and one write. An operation processing more than 50 records per minute approaches this limit. For operations with higher record volumes, use Cloud Datastore to store hashes instead.

Step 4: Map source fields to the target

With the condition in place, map fields from the source activity to the target activity in the transformation as usual. Records for which the condition returns false are skipped: no target mapping runs for those records.

Part 2: Within-run deduplication using cache

This pattern detects duplicate records within a single operation run. It is appropriate when a source dataset may contain the same record more than once (for example, a report that returns one row per transaction line, which can result in the same customer ID appearing multiple times).

Cache entries written in this pattern use the default 30-minute expiration and "project" scope (both defaults). The 30-minute window is long enough to span a single run, and the project scope is appropriate because the dedup state does not need to survive beyond the current run.

Step 1: Choose a dedup key

Identify the field or combination of fields that makes a record unique. If the source record has a primary key field such as id or order_id, use that directly. If uniqueness depends on a combination of fields, concatenate them with | separators.

Option A: Single unique ID field
```
Source.customer_id
```

Option B: Composite key

Source.order_id + "|" + Source.line_number

Step 2: Write the condition script

Open the transformation and add a condition script to the root-level node.

Replace "my_project_dedup_" with a stable prefix and replace Source.customer_id with the dedup key from Step 1.

// Build the dedup cache key
dedupeKey = "my_project_dedup_" + Source.customer_id;

// Check whether this key has been seen already in this run
existing = ReadCache(dedupeKey);

If(existing == null,
    // First occurrence: mark this key as seen and process the record
    WriteCache(dedupeKey, "1");
    true;
,
    // Duplicate: skip the record
    false;
);

Key points about this script:

ReadCache with no expiration or scope arguments uses the default 30-minute expiration and "project" scope, which is appropriate for within-run state.
WriteCache(dedupeKey, "1") stores a placeholder value. The value itself does not matter: the presence of the key in cache is the signal.
The first time a key is encountered, the condition returns true and the record is processed. Every subsequent occurrence of the same key returns false and is skipped.

Warning

ReadCache and WriteCache are subject to the same 100 calls per minute per organization limit as in Part 1. Each unique record in this pattern makes two cache calls (one read and one write on first occurrence). Duplicate records make one call (read only). For large datasets, a dictionary keyed on the dedup field is a higher-throughput alternative with no call-rate limit. See Populate and use a dictionary. Dictionaries are scoped to the operation thread and are not suitable for cross-run state.

Step 3: Map source fields to the target

Map fields from the source activity to the target activity as usual. Records for which the condition returns false are skipped.

Verify the integration

Verifying Part 1 (cross-run change detection)

Deploy and run the operation for the first time against your source data.
In the operation logs, confirm that all records were processed. On the first run, ReadCache returns null for every record (no stored hash exists yet), so every record's condition returns true.
Run the operation a second time without modifying any source records.
In the operation logs, confirm that no records were processed. All stored hashes now match the computed hashes, so every condition returns false.
Modify one source record, then run the operation a third time.
In the operation logs, confirm that exactly the modified record was processed and all others were skipped.
If records are unexpectedly skipped on the first run, confirm that the cache scope in both ReadCache and WriteCache is "env" and that the cache key expression is identical in both calls. A mismatch in scope or key name causes ReadCache to always return null, which means changes are always detected but hashes are written to an unreachable key.

Verifying Part 2 (within-run deduplication)

Prepare a source dataset that contains at least two records with the same dedup key value.
Deploy and run the operation.
In the operation logs, confirm that only one record was written to the target for each unique dedup key, and that duplicate records were skipped.
Run the operation a second time using the same source data.
Confirm that the first occurrence of each dedup key is processed again. The 30-minute default cache expiration means dedup state does not carry over between runs: each run starts fresh.

Note

If you run the operation twice within 30 minutes, cache entries from the first run are still live. In that case, the second run skips all records as duplicates, which is expected behavior. Wait for cache to expire before re-running, or use a different dedup key prefix to isolate test runs.
If duplicates are not being skipped, verify that the dedup key expression produces a stable, non-empty value for every record. A key that evaluates to null or an empty string causes every record to appear unique.