Amazon S3 List Objects activity

Introduction

An Amazon S3 List Objects activity, using its Amazon S3 connection, returns some or all (up to 100,000) of the objects in an Amazon S3 bucket and is intended to be used as a target to consume data in an operation.

Create an Amazon S3 List Objects activity

An instance of an Amazon S3 List Objects activity is created from an Amazon S3 connection using its List Objects activity type.

To create an instance of an activity, drag the activity type to the design canvas or copy the activity type and paste it on the design canvas. For details, see Creating an activity instance in Component reuse.

An existing Amazon S3 List Objects activity can be edited from these locations:

The design canvas (see Component actions menu in Design canvas).
The project pane's Components tab (see Component actions menu in Project pane Components tab).

Configure an Amazon S3 List Objects activity

Follow these steps to configure an Amazon S3 List Objects activity:

Step 1: Enter a name and select a bucket
Provide a name for the activity and select a bucket to be used.
Step 2: Review the data schemas
Any request or response schemas generated from the endpoint are displayed.

Step 1: Enter a name and select a bucket

In this step, provide a name for the activity and select a bucket to be used.

The interface used in this step of the configuration of an Amazon S3 List Objects activity depends on the Bucket Options selected:

Use lists: This option will populate a list of existing buckets to choose from.
Use text inputs: For this option, you will manually enter the name of the Amazon S3 bucket.

Use lists

If Use Lists is selected for Bucket Options, this interface is presented:

Amazon S3 List Objects activity configuration step 1

Name: Enter a name to identify the activity. The name must be unique for each Amazon S3 List Objects activity and must not contain forward slashes (/) or colons (:).
Select a Bucket: From the populated list, select the bucket containing the objects.
Optional Settings: Click to expand additional optional settings:
- Object Prefix: Enter a prefix to limit the response to keys that begin with the specified prefix.
  - Exclude Prefix Exact Match from Results: Select to exclude exact matches for the prefix specified in Object Prefix.
- Delimiter: Enter a character to use as a delimiter (maximum of 1 character).
- Max Keys: Enter the maximum number of keys returned to be returned in the response. By default the activity returns up to 1,000 key names. The maximum is 100,000 keys.
- Start After: Enter a key to return a response that begins after the specified key. The maximum value is 1,024 characters.
- AWS Region (Overrides Bucket's Region): Select an Amazon S3 Region.

Use text inputs

If Use Text Inputs is selected for Bucket Options, this interface is presented:

Amazon S3 List Objects activity use text inputs option

Name: Enter a name to identify the activity. The name must be unique for each Amazon S3 List Objects activity and must not contain forward slashes (/) or colons (:).
Bucket Name: Enter the name of the AmazonS3 bucket containing the objects.
Optional Settings: Click to expand additional optional settings:
- Object Prefix: Enter a prefix to limit the response to keys that begin with the specified prefix.
  - Exclude Prefix Exact Match from Results: Select to exclude exact matches for the prefix specified in Object Prefix.
- Delimiter: Enter a character to use as a delimiter (maximum of 1 character).
- Max Keys: Enter the maximum number of keys to be returned in the response. By default the activity returns 1,000 keys. The maximum is 100,000 keys.
- Start After: Enter a key to return a response that begins after the specified key. The maximum value is 1,024 characters.
- AWS Region (Overrides Bucket's Region): Select an Amazon S3 Region.
Save & Exit: If enabled, click to save the configuration for this step and close the activity configuration.
Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.
Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Step 2: Review the data schemas

Any request or response schemas generated from the endpoint are displayed. Each user interface element of this step is described below.

Amazon S3 List Objects activity configuration step 2

Data Schemas: The request and response schemas for this activity are displayed. These data schemas are inherited by adjacent transformations and are displayed again duringtransformation mapping.

The Amazon S3 connector uses the AWS SDK for Java version 1.12.770. Refer to the API documentation for information on the schema nodes and fields.

The request and response data schemas consist of these nodes and fields:

Response Schema Field	Description
keyCount	The number of keys returned
List	The list of keys, results returned in UTF-8 binary order
key	String with the key of the initiated multipart upload
size	Size
ETag	The entity tag identifying the new object using getETag method
lastModified	Datetime value of the Last-Modified header, indicating the date and time at which Amazon S3 last recorded a modification to the associated object, using the getLastModified method
error	Container for all error elements
errorCode	A string that uniquely identifies an error condition
message	A generic description of the error condition

Refresh: Click the refresh icon or the word Refresh to regenerate schemas from the Amazon S3 endpoint. This action also regenerates a schema in other locations throughout the project where the same schema is referenced, such as in an adjacent transformation.
Back: Click to temporarily store the configuration for this step and return to the previous step.
Finished: Click to save the configuration for all steps and close the activity configuration.
Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Next steps

After configuring an Amazon S3 List Objects activity, complete the configuration of the operation by adding and configuring other activities, transformations, or scripts as operation steps. You can also configure the operation settings, which include the ability to chain operations together that are in the same or different workflows.

Menu actions for an activity are accessible from the project pane and the design canvas. For details, see Activity actions menu in Connector basics.

Amazon S3 List Objects activities can be used as a target with these operation patterns:

Transformation pattern
Two-target archive pattern (as the first source only)
Two-target HTTP archive pattern (as the first source only)

To use the activity with scripting functions, write the data to a temporary location and then use that temporary location in the scripting function.

When ready, deploy and run the operation and validate behavior by checking the operation logs.