FTP Read activity

Introduction

An FTP Read activity reads data from an FTP endpoint and is intended to be used as a source to provide data to an operation or to be called in a script. After configuring an FTP connection, you can configure as many FTP activities as you like for each FTP connection.

Create an FTP activity

An instance of an activity is created from a connection using an activity type.

To create an instance of an activity, drag the activity type to the design canvas or copy the activity type and paste it on the design canvas. For details, see Creating an activity instance in Component reuse.

An existing activity can be edited from these locations:

• The design canvas (see Component actions menu in Design canvas).
• The project pane's Components tab (see Component actions menu in Project pane Components tab).

Configure an FTP Read activity

Follow these steps to configure an FTP Read activity:

• Step 1: Enter a name, provide a response schema, and specify settings
• Step 2: Review the data schemas

Step 1: Enter a name, provide a response schema, and specify settings

• Name: Enter a name to use to identify the FTP Read activity. The name must be unique for each FTP Read activity and must not contain forward slashes (/) or colons (:).

• Provide Response Schema: The response schema defines the structure of data that is used by the FTP Read activity. Whether a response schema is required depends on if the activity is used as the source of a transformation (see Schema usage). For instructions on completing this section of activity configuration, refer to Schemas defined in an activity.

• Path: Enter a single valid path. A path should point to an existing directory on the FTP server. For example, /inbound.

• Get Files: Enter the name of the file(s) that you want to read from within the path entered above. This field may also include an asterisk (*) to use as a wildcard (for example, *.dat or *.*) or a question mark (?) to match exactly one character (for example, file?.txt). The use of wildcards is separate from the Use Regular Expression option described below. If specifying data in ZIP format to be automatically decompressed and read, see the Compression options described below.

  Tip

  If you are missing known files or if the names of files have extra information attached to them after retrieving them from the endpoint, set Jitterbit.source.FTP.enable_regex_parser to true upstream of an FTP Read activity.

• Optional Settings: Click to expand additional optional settings:

  

  • File Filter Options: These options affect how the input entered in the Get Files field above is interpreted:

    • Ignore Case: Select this checkbox if you have entered input in the Get Files field above that should be interpreted as case-insensitive.

    • Use Regular Expression: Select this checkbox if you have entered input in the Get Files field above that should be interpreted as a regular expression. This option is separate from the use of wildcards described in Get Files above.

  • List Only File Names: Check this box if you want to list only file names (using the FTP command NLST instead of LIST). Try this if Harmony cannot parse the FTP directory listing sent back by the FTP server.

    Tip

    If you are missing known files or if the names of files have extra information attached to them after retrieving them from the endpoint, set Jitterbit.source.FTP.enable_regex_parser to true upstream of an FTP Read activity.

  • Character Encoding: Specify character encoding to define a specific character encoding for the source file. If you leave the field blank, Harmony tries to detect the encoding. If you want to overwrite the default, you can choose from any of the encoding types listed in Supported character encodings.

  • Success/Error Folder: These options are used to archive a copy of all files processed. This can be used in place of the After Processing option to Rename File on the file source to allow you to write to a separate location rather than to a sub-folder in the same location.

    • Use Success Folder: Select this checkbox to write a file to the specified folder after successful processing on the primary target path. If writing a file to multiple paths, the operation is successful only if all paths are successful. If this checkbox is selected, also provide the Success Folder in Absolute Path on the private agent machine.

    • Use Error Folder: Select this checkbox to write a file to the specified folder after unsuccessful processing. If at least one path is unsuccessful, then the file is written to this folder. If this checkbox is selected, also provide the Error Folder in Absolute Path on the private agent machine.

      Caution

      A file is written only if the failure is due to errors processing the FTP Read activity. A file is not written if, for example, the FTP connection is unsuccessful, causing the entire operation to fail.

    Important

    Absolute paths for the success and error folders are located on the private agent, not on the FTP server.

    When using filename keywords in these fields, the keywords are not resolved. For a workaround, see Limitation and workaround under Filename keywords. Using asterisks (*) and other typically unsupported file name characters will cause the operation to fail.

    The file(s) are named as specified in the Get Files field above, with a trailing underscore and a timestamp added. If there is already a file with the same name in that folder, an additional underscore and a counter are added.

    For example, if a file named data.csv is successfully processed, a copy named data.csv_2018-01-01_12-00-00-000 could be written to the folder specified as the success folder. If such a file already exists from a previous run, a new file data.csv_2018-01-01_12-00-00-000_1 would be added. In a similar fashion, if an error folder is specified, any files that fail are written to a file in the error folder.

  • After Processing: If you want to perform further actions after processing the data, these options are available:

    • Delete File: Select this checkbox if you want files that have been successfully processed to be deleted. When this option is selected, the Rename File option below is automatically populated with /\delete/\ to indicate that processed files will be deleted and as such, renaming a file does not apply. Clearing Delete File also clears the Rename File field.

    • Rename File: Enter a name if you want to rename files after processing. Any of the filename keywords are available to generate unique filenames for the source data after processing. Note the Rename File field is unavailable if Delete File above is selected. In addition, toggling Delete File above clears any input entered in the Rename File field.

      Tip

      If you want to archive the file source to a separate location on a private agent, also see the Success/Error Folder option on the FTP Write activity configuration.

  • Trigger File: Available only for use with private agents where local files are enabled, a trigger file is a file that must be present before a source file is retrieved by an operation for processing. The presence of a trigger file indicates that a source file is ready to be read and is not still in the process of being written by a previous operation.

    • Use Trigger: Select this checkbox if you want the system to look for a trigger file before processing source files. Note, when using a trigger file:

      • The operation polls, looking for the trigger file. Polling time interval is fixed at one second.

      • The trigger file is deleted as processing starts.

      • If the trigger file is not found, the operation keeps attempting to find the trigger file until the operation times out.

      • Setting a trigger file blocks any operation or script that uses the source until either the trigger file condition is satisfied or the operation times out.

    • Absolute File Path: Enter the absolute file path on the private agent machine where the trigger file resides. Select a file that meets these criteria:

      • The file can be of any type and may contain anything as its contents; only the presence of the file defines the trigger file behavior.

      • The use of wildcards is not permitted.

      • The trigger file cannot be the same file as a source file being processed.

      Note

      When using filename keywords in this field, the keywords are not resolved. For a workaround, see Limitation and workaround under Filename keywords.

  • Ignore Lines: If there are headers and/or footers in the source data, enter the number of lines to ignore in the First and Last fields.

    Note

    This setting does not take effect when an FTP Read activity is referenced from a script using the ReadFile or WriteFile functions.

  • Compression: If the source data is in a compressed format, select The Source Data Is Compressed. These options become available:

    • Format: Use the dropdown to select the format of the compressed data. Currently, only ZIP format is supported.

    • Password: If the archive is encrypted, enter the password.

      Note

      Harmony supports reading only from the root directory of an encrypted archive.

  • Transfer Type: If the files are ASCII-based, choose the ASCII transfer type. For all other file types, choose Binary. Binary is the default and usually works best.

• Save & Exit: If enabled, click to save the configuration for this step and close the activity configuration.

• Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Step 2: Review the data schemas

• Data Schema: If provided during activity configuration, the response data schema is displayed. If the operation uses a transformation, the data schemas are displayed again later during the transformation mapping process, where you can map to target fields using source objects, scripts, variables, custom values, and more. You can also define schemas directly in a transformation.

• Add Plugin(s): Plugins are Jitterbit- or user-provided applications that extend Harmony's native capabilities. To apply a plugin to the activity, click to expand this section and select the checkbox next to the plugin to be used. For additional instructions on using plugins, including details on setting any required variables used by the plugin, see Plugins added to an activity.

• Back: Click to temporarily store the configuration for this step and return to the previous step.

• Finished: Click to save the configuration for all steps and close the activity configuration.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Next steps

After configuring an FTP Read activity, you can use it within an operation or script as described below.

Complete the operation

After configuring an FTP Read activity, complete the configuration of the operation by adding and configuring other activities, transformations, or scripts as operation steps. You can also configure an operation's operation settings, which include the ability to chain operations together that are in the same or different workflows.

Once an FTP Read activity has been created, menu actions for that activity are accessible from the project pane in either the Workflows or the Components tabs, and from the design canvas. See Activity actions menu for details.

FTP Read activities can be used as a source with these operation patterns:

• Archive pattern
• Transformation pattern
• Two-target archive pattern (as the first source only)
• Two-target HTTP archive pattern (as the first source only)
• Two-transformation pattern (as the first source only)
• Salesforce bulk target pattern

Other patterns are not valid using FTP Read activities. See the validation patterns on the Operation validity page.

When ready, deploy and run the operation and validate behavior by checking the operation logs.

Using FTP activities in scripts

FTP Read activities can also be referenced in a script for use with script functions that use a sourceId as a parameter, including these:

Jitterbit Script Functions

• ArchiveFile
• Base64EncodeFile
• DeleteFile
• DeleteFiles
• DirList
• FileList
• ReadFile

JavaScript Jitterbit Functions

• Jitterbit.ReadFile

For more details on referencing activities in scripts, see Endpoints in Jitterbit Script or Endpoints in JavaScript.