Skip to Content

Persist data using project variables, cloud caching, or Temporary Storage in Jitterbit Integration Studio

Introduction

Multiple methods can be used for storing data in the Harmony cloud, such as using project variables, using cloud caching functions, or using temporary storage.

Project variables

Project variables are preinitialized static variables that can be thought of as project constants. They can be edited from either Integration Studio (see Project variables) or the Management Console (see Projects).

One example use of project variables is for endpoint credentials. By using project variables, different endpoint environments (which usually have different credentials) can be applied to different Harmony environments and edited through the Management Console. This enables a business process where a user with Management Console rights can change credentials without requiring access to Integration Studio and the project designer.

A second example is to use project variables to hold flags used by integration logic that can customize the behavior of the integration. If a single project is developed but used for different endpoints, then a boolean project variable (such as Send_PO_Number) could be checked by the transformation's logic for the PO Number field. If the project variable's value is false, then the UnMap function could be used to "turn off" that field and not include it in a relevant transformation.

Cloud caching functions

The cloud caching functions ReadCache and WriteCache are used to assign data spaces that are available across projects and across environments. A cached value is visible to all operations running in the same scope until it expires, regardless of how that operation was started or which agent it runs on. By caching data in Harmony, rather than relying on local or agent-specific data stores such as Temporary Storage, data can be shared between separate operations and across projects.

These are additional uses of cloud caching:

  • Data can be shared between asynchronous operations within a project.
  • Errors that are generated across different operations could be stored to a common cache. By accumulating operation results in this manner, more comprehensive alerts can be built.
  • Login tokens can be shared across operations.

Temporary Storage

Temporary Storage endpoints are frequently used in operations on both cloud and private agents. These are distinct from Local Storage endpoints, which can only be used on private agents.

When using a Temporary Storage endpoint, temporary files are written to and read from the default operating system's temp directory on the agent that is performing the work:

  • In the case of a single private agent, the temporary storage directory is that private agent server's default temp directory.
  • If you are using more than one private agent, clustered in a private agent group, the temporary storage directory is the default temp directory of the specific private agent server doing the work.
  • As cloud agents are clustered in a cloud agent group, the temporary storage directory is the default temp directory of the specific cloud agent server doing the work.

In a clustered agent group (private or cloud agents), as long as the operations using the temporary storage are linked (chained) together, then all the temporary storage reads and writes will happen on the same agent server. However, if Chain A writes to its temp storage myfile and Chain B reads from its temp storage myfile, and the two chains are not themselves chained to one another, the temporary storage read action in Chain B may not be read from the same agent that was written to by Chain A.

When using temporary storage, keep these guidelines in mind:

  • When using private agents, to make your project upgrade-proof, use temporary storage in such a way that moving from a single private agent to a multiple-agent agent group does not require refactoring.

  • When using a clustered agent group (private or cloud agents), to ensure that the agent server where temporary storage is written is the same server where temporary storage will be read from, make sure that any references to the temporary storage Read and Write activities are in the same operation chain.

  • Temporary storage on private agents is deleted after 24 hours by default by the [Jitterbit file cleanup service]/agent/cleanuprules/). The cleanup service frequency can be configured through the private agent configuration file under the [FileCleanup] section. However, on cloud agents, temporary files may be deleted immediately.

  • Cloud agents have a temporary storage file size limit of 50 GB per file. Temporary files larger than 50 GB are possible only when using private agents.

  • When writing to temporary storage, the default is to overwrite files. This can be changed with the Append to File checkbox in a Temporary Storage Write activity. Usually this then requires that after the source is read that the file be deleted or archived. A simple way to do this is to use the post-processing options Delete File or Rename File in a Temporary Storage Read activity.

  • Filename keywords are available that can be used when creating a filename.

  • A file in temporary storage can be read by building a script with the ReadFile function. For example: ReadFile("<TAG>activity:tempstorage/Temporary Storage/tempstorage_read/Read</TAG>"). Bear in mind that this works reliably only if there is a single private agent.

In some cases, it may be advantageous to use a Variable endpoint instead of a Temporary Storage endpoint. See the section Variable versus Temporary Storage in Global variable versus Temporary Storage for a comparison of these two different approaches and for recommendations on when each is appropriate.