Skip to Content

Turn your connections into holiday cash with our new Customer Referral Program! Learn more

✕

Temporary storage considerations in Integration Studio

There are multiple solutions offered by Integration Studio to users who require temporary storage in their integrations. This page provides considerations for choosing which solution to use, an endpoint using either the Variable connector or the Temporary Storage connector.

Variable endpoint

Variable endpoints (Read and Write activities, not to be confused with scripting global variables) are easy to code and reduce complexity, as described later on this page. However, they have certain limitations.

For a scenario where an integration is working with tiny data sets — typical of web service requests and responses, or small files of a few hundred records — we suggest using a Variable endpoint.

When the data set is in the megabyte range, the Variable endpoint becomes slower than the equivalent Temporary Storage endpoint. This starts to happen when the data becomes over 4 MB in size.

When the data set is in the larger multi-megabyte range, there is a risk of data truncation. We recommend a limit of 50 MB to be conservative and prevent any risk of truncation occurring.

Using Variable endpoints in asynchronous operations is a use case that requires special consideration. There is a limit of 7 KB on the size of a data set used in a Variable endpoint that is used in an asynchronous operation. In this scenario, exceeding that limit can result in truncation. See the RunOperation() function for a description of calling an asynchronous operation.

Variable endpoints increase reuse and reduce complexity

Using a Variable endpoint for tiny data sets can increase reuse and reduce complexity. For example, when building operations chained with operation actions, each operation can have activities that function as sources (Read activities) and targets (Write activities). Instead of building individual source or target combinations for each operation, it is easy to use a common Variable target and source.

To increase reusability and standardization, you can build a reusable script that logs the content of the variable. This approach can also be accomplished using temporary storage, but additional scripting is needed to initialize the path and filename.

When using a Variable endpoint, its scope is the chain — the thread — of operations. Thus, Variable endpoint values are unique to a particular thread, and are destroyed when the thread is finished. This is not the case with a Temporary Storage endpoint; as a result, it requires more handling to ensure uniqueness. The best practice is to initialize a GUID at the start of an operation chain and then pass that GUID to each of the temporary storage file names in the chain, as described in Persist data for later processing using Temporary Storage. (Although that document is for Design Studio, the same concepts can be applied to Integration Studio.)

When performing operation unit testing, it is helpful to load test data. Using a Variable source or target makes this simple: you add a pre-operation script to write the test data to a target:

  $memory = "a,b,c";

In contrast, writing data to a Temporary Storage endpoint looks like this:

  WriteFile("<TAG>activity:tempstorage/Temporary Storage/tempstorage_write/Write</TAG>", "a,b,c");

  FlushFile("<TAG>activity:tempstorage/Temporary Storage/tempstorage_write/Write</TAG>");

Likewise, reading data is simpler with a Variable endpoint:

  myLocalVar= $memory;

In contrast, this is how you read data from a Temporary Storage endpoint:

  myLocalVar = ReadFile("<TAG>activity:tempstorage/Temporary Storage/tempstorage_read/Read</TAG>");

In summary, using Variable endpoints for reading, writing, and logging operation input and output is straightforward, but great caution needs to be given to make sure the data is appropriately sized.

Temporary Storage endpoint

Larger data sets, such as those with record counts in the thousands, should be handled using Temporary Storage endpoints.

Unlike Variable endpoints, there is no degradation in performance or truncation when using Temporary Storage endpoints, even with very large data sets. However, using Temporary Storage endpoints may require additional scripting. By using Temporary Storage endpoints, you are not able to take advantage of the reuse and simplicity of Variable endpoints, as described later on this page.

Note that cloud agents have a Temporary Storage endpoint file size limit of 50 GB per file. Those who need to create temporary files larger than 50 GB will require a private agent.