Skip to Content

Apache Parquet connection

Introduction

An Apache Parquet connection, created using the Apache Parquet connector, establishes access to Apache Parquet. These activities, once configured, interact with Apache Parquet through the connection.

Create or edit an Apache Parquet connection

A new Apache Parquet connection is created using the Apache Parquet connector from one of these locations:

An existing Apache Parquet connection can be edited from these locations:

Configure an Apache Parquet connection

Each user interface element of the Apache Parquet connection configuration screen is described below.

Tip

Fields with a variable icon support using global variables, project variables, and Jitterbit variables. Begin either by typing an open square bracket [ into the field or by clicking the variable icon to display a list of the existing variables to choose from.

  • Connection Name: Enter a name to use to identify the connection. The name must be unique for each Apache Parquet connection and must not contain forward slashes (/) or colons (:). This name is also used to identify the Apache Parquet endpoint, which refers to both a specific connection and its activities.

  • URI: Enter the URI.

  • Connection Type: Select the connection type, one of:

    • Auto

    • Local

    • Amazon S3

    • Azure Blob Storage

    • Azure Data Lake Storage Gen1

    • Azure Data Lake Storage Gen2

    • Azure Data Lake Storage Gen2 SSL

    • Azure Files

    • Box

    • Dropbox

    • FTP

    • FTPS

    • Google Cloud Storage

    • Google Drive

    • HDFS

    • HDFS Secure

    • HTTP

    • HTTPS

    • IBM Object Storage Source

    • OneDrive

    • OneLake

    • Oracle Cloud Storage

    • SFTP

    • SharePoint REST

    • SharePoint SOAP

    For more information on connection types, see the Apache Parquet connection details page.

  • Data Model: Select the data model, one of:

    • None

    • Document

    • FlattenedDocuments

    • Relational

  • OAuth Grant Type: Select the OAuth grant type, one of:

    • CODE

    • CLIENT

    • PASSWORD

  • Aggregate Files: When selected, specifies to use aggregate files.

  • Use Proxy Settings: Select to use private agent proxy settings.

  • Advanced Configurations: When selected, shows the Advanced Configurations Properties table, which is used to add additional configuration properties as key-value pairs.

    • Add: Adds a row to the table. Enter a key-value pair for each property.
    • Submit: Saves the key-value pair to the table. Visible when hovering on a row in edit mode.
    • Discard: Discards a new row or any edits made to an existing row. Visible when hovering on a row in edit mode.
    • Edit: Allows you to edit an existing key-value pair. Visible when hovering on an existing row.
    • Delete: Deletes the row from the table. Visible when hovering on an existing row.
    • Clear All: Deletes all rows from the table.

    Important

    Fields in the Advanced Configurations table display the variable icon only in edit mode. For these fields' variable values to be populated at runtime, the agent version must be at least 10.75 / 11.13.

  • Test: Click to verify the connection using the specified configuration. When the connection is tested, the latest version of the connector is downloaded by the agent(s) in the agent group associated with the current environment. This connector supports suspending the download of the latest connector version by using the Disable Auto Connector Update organization policy.

  • Save Changes: Click to save and close the connection configuration.

  • Discard Changes: After making changes to a new or existing configuration, click to close the configuration without saving. A message asks you to confirm that you want to discard changes.

  • Delete: After opening an existing connection configuration, click to permanently delete the connection from the project and close the configuration (see Component dependencies, deletion, and removal). A message asks you to confirm that you want to delete the connection.

Configure OAuth connections

This connector supports a connection using OAuth by using specific configuration properties. After selecting an auth scheme that includes Initiate OAuth, you will be able to specify the configuration in the fields that are displayed.

Note

Additional IDs, secrets, scopes, and other authentication tokens may be required in order to successfully complete the OAuth authentication. Check the endpoint documentation for additional information on their requirements.

For this connector, two scenarios are supported. In either scenario, you will need to use a third-party tool (such as Postman or a similar tool) to obtain the required access token or tokens from the endpoint involved. Both of these scenarios work only for single-agent groups. These are agent groups that consist of a single agent.

  • Scenario 1: Using an OAuth access token that expires after a set period of time.

    With this configuration, you supply an OAuth access token that lasts for a set period of time, as determined by the particular endpoint. The connector will use the token to make the connection. The connection, however, will no longer work once the access token expires. You would configure the connector to use the token by setting these properties:

    Configuration Property Value
    Initiate OAuth OFF
    OAuth Access Token Set to the value of the OAuth access token obtained from the endpoint

  • Scenario 2: Using an OAuth access token combined with an OAuth refresh token to enable automatic refresh of access.

    With this configuration, the connector will use a refresh token to obtain a new access token once the current access token has expired. This is generally the best method of configuring OAuth.

    You supply an OAuth access token and an OAuth refresh token, as determined by the particular endpoint. You would configure the connector to use these tokens by setting these properties:

    Configuration Property Value
    Initiate OAuth REFRESH
    OAuth Access Token Set to the value of the OAuth access token obtained from the endpoint
    OAuth Refresh Token Set to the value of the OAuth refresh token obtained from the endpoint

Note

If using multiple OAuth-based Apache Parquet connections on the same private agent, set the configuration property OAuthSettingsLocation to a unique writeable agent path for each connection. In the jitterbit.conf file for the agent, set EnableLocalFileLocation=true. If you don't perform these two steps, the multiple connections will use the same default path and overwrite each other.

Next steps

After an Apache Parquet connector connection has been created, you place an activity type on the design canvas to create activity instances to be used either as sources (to provide data in an operation) or as targets (to consume data in an operation).

Menu actions for a connection and its activity types are accessible from the project pane and design component palette. For details, see Actions menus in Connector basics.

These activity types are available:

  • Query: Retrieves records from a table at Apache Parquet and is intended to be used as a source in an operation.

  • Create: Inserts a record into a table at Apache Parquet and is intended to be used as a target in an operation.

  • Execute: Executes a procedure at Apache Parquet and is intended to be used as a target in an operation.