Apache Kafka Get Messages activity
Introduction
An Apache Kafka Get Messages activity, using its Apache Kafka connection, retrieves streams of data from a topic in an Apache Kafka cluster and is intended to be used as a source to provide data in an operation.
Create an Apache Kafka Get Messages activity
An instance of an Apache Kafka Get Messages activity is created from an Apache Kafka connection using its Get Messages activity type.
To create an instance of an activity, drag the activity type to the design canvas or copy the activity type and paste it on the design canvas. For details, see Creating an activity instance in Component reuse.
An existing Apache Kafka Get Messages activity can be edited from these locations:
- The design canvas (see Component actions menu in Design canvas).
- The project pane's Components tab (see Component actions menu in Project pane Components tab).
Configure an Apache Kafka Get Messages activity
Follow these steps to configure an Apache Kafka Get Messages activity:
-
Step 1: Enter a name
Enter a name for the activity. -
Step 2: Review the data schemas
Any request or response schemas generated from the endpoint are displayed.
Step 1: Enter a name
In this step, enter a name for the activity. Each user interface element of this step is described below.
-
Name: Enter a name to identify the activity. The name must be unique for each Apache Kafka Get Messages activity and must not contain forward slashes
/or colons:. -
Save & Exit: If enabled, click to save the configuration for this step and close the activity configuration.
-
Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.
-
Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.
Step 2: Review the data schemas
Any request or response schemas generated from the endpoint are displayed. Each user interface element of this step is described below.
- Data Schemas: These data schemas are inherited by adjacent transformations and are displayed again during transformation mapping.
The Apache Kafka connector uses the Apache Kafka Core API. Refer to the API documentation for information on the schema nodes and fields.
The request and response data schemas consist of these nodes and fields:
| Request Schema Node/Field | Description |
|---|---|
getMessagesRequest | Node representing the activity request |
consumerGroup | String containing the consumer group to be used to consume data from the topic |
sessionTimeoutMS | Integer of the amount of time (in milliseconds) to detect client failures when using Apache Kafka's group management facility |
maxPollRecords | Integer of the maximum number of records returned in a single call |
topic | String with the name of the topic containing the messages |
pollDurationSec | Integer of the time (in seconds) after which an empty record set will be returned if no messages are retrieved |
| Response Schema Node/Field | Description |
|---|---|
getMessagesResponse | Node representing the activity response |
message | Node representing the retrieved message |
item | Node representing the items within a message |
key | String containing the key of the retrieved message |
value | String containing the value of the retrieved message |
offset | String containing the offset that the message is assigned to |
topic | String with the name of the topic the message belongs to |
-
Refresh: Click the refresh icon or the word Refresh to regenerate schemas from the Apache Kafka endpoint. This action also regenerates a schema in other locations throughout the project where the same schema is referenced, such as in an adjacent transformation.
-
Back: Click to temporarily store the configuration for this step and return to the previous step.
-
Finished: Click to save the configuration for all steps and close the activity configuration.
-
Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.
Next steps
After configuring an Apache Kafka Get Messages activity, complete the configuration of the operation by adding and configuring other activities, transformations, or scripts as operation steps. You can also configure the operation settings, which include the ability to chain operations together that are in the same or different workflows.
Menu actions for an activity are accessible from the project pane and the design canvas. For details, see Activity actions menu in Connector basics.
Apache Kafka Get Messages activities can be used as a source with these operation patterns:
- Transformation pattern
- Two-target archive pattern (as the first source only)
- Two-target HTTP archive pattern (as the first source only)
- Two-transformation pattern (as the first or second source)
To use the activity with scripting functions, write the data to a temporary location and then use that temporary location in the scripting function.
When ready, deploy and run the operation and validate behavior by checking the operation logs.