Temporary Storage in Jitterbit Design Studio
Temporary storage is a built-in location, either as a target or source type, that can be used as a temporary repository of files. A temporary storage location is referenced by a path which points to a folder in a temporary directory on the private agent or cloud agent. By default, a directory under the operating system's temporary folder will be used.
Any appropriate file type can be used as storage. Commonly used are text (.txt
), CSV (.csv
), XML (.xml
), and JSON (.json
).
A temporary storage location is referred to as either a temporary storage target or source, depending on how it was created.
Default location for Temporary Storage
For private agents, this default location can be specified in the private agent configuration file jitterbit.conf
by a setting in the [OperationEngine]
section:
TemporaryStorageRootDirectory=<absolute-path-to-existing-directory>
Substitute for <absolute-path-to-existing-directory>
. For example:
TemporaryStorageRootDirectory=/tmp/tmpstore
If a file has been written to temporary storage using a Temporary Storage target with a particular path, the same file can be retrieved by a Temporary Storage source using the same path. However, if multiple agents or operations are used, the agent could change and the file no longer be available.
Temporary Storage is typically located:
- Windows agents:
C:\Windows\Temp\jitterbit\TemporaryStorage
- Linux agents:
/tmp/jitterbit/TemporaryStorage
See Edit the configuration file (jitterbit.conf) for details.
Private agents
For private agents, when using temporary storage be aware that:
- Temporary Storage is written to the operating system's default temp directory (or the directory specified in agent configuration) on the agent that is performing the work. In the case of a single private agent, then it is that agent's server host's default temp directory. If the agent group is running more than one private agent, then it is the temp directory on the server host for the particular agent that is doing the work.
- A temporary storage location needs to be used very soon after it is created in order for additional processing to take place on the same private agent where the temporary storage is located.
- Temporary files should only be read by the operation chain that writes them. If multiple private agents are assigned to an agent group, operations will run on any one of the agents in the group, so temporary files cannot be written by one operation chain and then read by a different chain unless the chains are linked. In a clustered agent environment (private or cloud agents), as long as the operations using the temporary storage are linked (chained) together, then all the temp file reads and writes will happen on the same server host. See Persisting data for later processing using Temporary Storage for examples and details.
- By default, temporary storage on a private agent is deleted after 24 hours by the Jitterbit file cleanup service.This can be changed for the Jitterbit cleanup service in the configuration file
jitterbit.conf
in the[FileCleanup]
section by the settingFrequencyInHours
.
Cloud agents
For cloud agents, when using temporary storage, be aware of these restrictions:
- Cloud agents that are version 10.10 or higher have a temporary storage file size limit of 50 GB per file. Those who need to create temporary files larger than 50 GB are encouraged to use private agents.
- When an operation writes to a temporary file, by default it is written to disk storage of the individual server within the cloud agent group that is assigned the work.
- A temporary storage location needs to be used very soon after it is created in order for additional processing to take place on the same cloud agent where the temporary storage is located.
- Temporary files should only be read by the operation chain that writes them. Since multiple cloud agents are assigned to a cloud agent group, operations will run on any one of several agents in the group, so temporary files cannot be written by one operation chain and then read by a different chain unless the chains are linked. In a clustered agent environment (private or cloud agents), as long as the operations using the temporary storage are linked (chained) together, then all the temp file reads and writes will happen on the same server host. See Persisting data for later processing using Temporary Storage for examples and details.
- Unlike private agents, the Jitterbit clean-up service runs continuously on cloud agents and may delete temporary storage immediately after the operation chain finishes processing.