Skip to Content

Listening service on Jitterbit private agents

Introduction

The listening service is a feature of private agents that can run an integration operation when an event happens on an endpoint, and can load-balance the processing of operations across a cluster of agents.

The service works only on private agent groups, and only with the following connectors and activities:

Prerequisites

To use the listening service, you must have the following:

  • Private agents version 10.78 / 11.8 or later.

  • TCP ports 5701 and 5801 open on all agent hosts.

  • All agents on the same network.

  • At least 3, or \((N / 2) + 1\) of a total of \(N\) agents in the agent group must be running.

Enable the listening service

The listening service is disabled by default. You must enable the listening service on all private agents in a group, and on both the project's operation and activity.

Enable agents

To enable the service, follow these steps on all private agents in the group:

  1. Edit JITTERBIT_HOME/Resources/jitterbit-agent-config.properties.

  2. Add these lines:

    1
2
3
    agent.sdk_framework.listener.enabled=true
agent.sdk_framework.listeners.matchEventsQueueToAvailableCores=true
# agent.sdk_framework.listeners.eventsQueue=N

    • Line 1: Enable the service.

    • Line 2: Sets the size of the events queue automatically.

    • Line 3: (Optional) Uncomment to set the size of the events queue manually, where \(N\) is less than or equal to the number of processor cores on the agent host. (If used, comment out line 2.)

  3. Restart the agent.

Enable operations and activities

By default, event listening is disabled. To enable event listening on operations and activities, follow these steps:

  1. Open the project and find the operation that contains the listening activity. An activity's listening capability is shown by an icon in the top right, for enabled, or for not enabled.

  2. If the operation is not yet deployed, deploy it. After it is deployed, the Events Disabled toggle appears at the bottom of the operation:

    Example listening activity

  3. To enable event listening for the operation, click the Events Disabled toggle.

    Important

    Enabling or disabling event listening for an operation affects all listening-enabled activities in the operation.

Monitor the listening service

You can monitor the status of the listening service with the Management Console, or the REST API.

Monitor with the Management Console

Once an operation with a listener activity is deployed, the operation and its listening activities are shown in the Listeners tab of the Management Console Projects page:

projects listeners

Monitor with the REST API

You can send an HTTP GET request to the listening service to see status information. The following information endpoints are available:

Endpoint Description
localhost:46912/axis/v1/cdk/internal/leader Show the cluster leader agent.
localhost:46912/axis/v1/cdk/internal/cluster Show the cluster status.
localhost:46912/axis/v1/cdk/internal/starter/stats Show statistics.
localhost:46912/axis/v1/cdk/internal/members Show restore process.

Tip

To reset the statistics, run this command:

curl -XDELETE localhost:46912/axis/v1/cdk/internal/starter/stats

Examples

Show the cluster leader agent:

Command
curl --silent localhost:46912/axis/v1/cdk/internal/leader | python -m json.tool
Example response
{
    "AgentClusterMemberUUID": "67a77314-46c7-4243-9193-bc143ece5ed0",
    "OrgId": "20980",
    "AgentId": "281020",
    "AgentHost": "192.168.64.18",
    "AgentName": "Agent 1",
    "AgentGroupId": "171230",
    "AgentGroupName": "Agent Group A"
}

Show the cluster status:

Command
curl --silent localhost:46912/axis/v1/cdk/internal/cluster | python -m json.tool
Example response
{
    "clusterTime": 1725895381981,
    "currentNumberOfMembers": 2,
    "state": "ACTIVE",
    "hasMinimumSize": true,
    "minimumMembersRequired": 2
}

Show statistics:

curl --silent localhost:46912/axis/v1/cdk/internal/starter/stats | python -m json.tool
Example response
{
    "sdk-retryable-events-queue": 0,
    "sdk-inflight-messages-0": 0,
    "sdk-processing-events-queue-0": {
        "stats": null,
        "lastProcessedEvent": -1,
        "queueSize": 0
    }
}

Show members:

curl --silent localhost:46912/axis/v1/cdk/internal/members | python -m json.tool
Example response
[
    {
        "isLeader": "true",
        "minClusterSize": "2",
        "inetSocketAddress": "192.168.64.18",
        "uuid": "67a77314-46c7-4243-9193-bc143ece5ed0",
        "version": "5.0.5",
        "isLiteMember": "false",
        "leaderNodeState": "STARTED",
        "ipv4": "true",
        "inetAddress": "192.168.64.18",
        "port": "5701",
        "ipv6": "false",
        "host": "rocky89",
        "AgentName": "Agent 1",
        "AgentId": "281020"
    },
    {
        "isLeader": "false",
        "minClusterSize": "2",
        "inetSocketAddress": "192.168.64.25",
        "uuid": "efbfab6a-0326-43c0-a4f7-f14a26a36d25",
        "version": "5.0.5",
        "isLiteMember": "false",
        "ipv4": "true",
        "inetAddress": "192.168.64.25",
        "port": "5701",
        "ipv6": "false",
        "host": "192.168.64.25",
        "AgentName": "Agent 2",
        "AgentId": "281030"
    }
]

Enable persistence

Messages sent to an agent can be stored in a database. If the agent fails, the database provides a persistent store of messages that can be resent when the agent comes back online.

By default, the internal PostgreSQL database stores this information, but only for single-agent clusters. For clusters with more than one agent, you must manually enable persistence.

You can also use an external JDBC-enabled database or Redis server.

Multi-agent persistence (internal PostgreSQL)

To enable multi-agent persistence on the internal PostgreSQL database, follow these steps on all private agents in the group:

  1. Edit JITTERBIT_HOME/Resources/jitterbit-agent-config.properties.

  2. Add these lines:

    agent.sdk_framework.queueStore.enabled=true
agent.sdk_framework.queueStore.type=dbinternal
agent.sdk_framework.persistence.enabled=true
agent.sdk_framework.persistence.type=dbinternal

  3. Restart the agent.

Multi-agent persistence (external JDBC or Redis)

To enable multi-agent persistence using an external JDBC-enabled database or Redis server, follow these steps on all private agents in the group:

  1. Edit JITTERBIT_HOME/Resources/jitterbit-agent-config.properties.

  2. If the database uses a JDBC connection, add these lines:

    JDBC-enabled database persistent store settings (Example)
    agent.sdk_framework.queueStore.enabled=true
agent.sdk_framework.queueStore.type=db

agent.sdk_framework.persistence.enabled=true
agent.sdk_framework.persistence.type=db

agent.sdk_framework.datastore.db.url=jdbc:sqlserver://harmony:1433;database=cloud
agent.sdk_framework.datastore.db.user=sa
agent.sdk_framework.datastore.db.password=******
agent.sdk_framework.datastore.db.databaseName=cloud
agent.sdk_framework.datastore.db.dialect=org.hibernate.dialect.SQLServerDialect
agent.sdk_framework.datastore.db.driver_class=com.microsoft.sqlserver.jdbc.SQLServerDriver

    If the database is Redis, add these lines:

    Redis persistent store settings (Example)
    agent.sdk_framework.queueStore.enabled=true
agent.sdk_framework.queueStore.type=redis

agent.sdk_framework.persistence.enabled=true
agent.sdk_framework.persistence.type=redis

agent.sdk_framework.datastore.redis.url=redis://redis:6379

#Optional - pool configuration

agent.sdk_framework.datastore.redis.maxTotal=8
agent.sdk_framework.datastore.redis.maxIdle=8
agent.sdk_framework.datastore.redis.minIdle=0
agent.sdk_framework.datastore.redis.blockWhenExhausted=true
agent.sdk_framework.datastore.redis.maxWaitMillis=-1
agent.sdk_framework.datastore.redis.testOnBorrow=false
agent.sdk_framework.datastore.redis.testOnReturn=false
agent.sdk_framework.datastore.redis.jmxEnabled=true

  3. Restart the agent.

Important

The database user must have permissions to create tables, and to create, read, update, and delete data.

Troubleshooting

Cluster restore after agent failure

If an agent fails with persistence enabled, you can manually restore it using these steps:

  1. Edit JITTERBIT_HOME/Resources/jitterbit-agent-config.properties on the failed agent's host.

  2. Add this line:

    agent.sdk_framework.listener.running.mode=restore

  3. Start the agent.

  4. Use the REST API to query the status of the agent.

  5. When the status for leaderNodeState is RESTORED, stop the agent.

  6. Edit JITTERBIT_HOME/Resources/jitterbit-agent-config.properties again, and remove or comment out the line added in step 2.

  7. Start the agent.

Undelivered messages

The cluster message delivery retry mechanism deletes undelivered messages after a period. To change the period, or to suppress message deletion, follow these steps:

  1. Edit JITTERBIT_HOME/Resources/jitterbit-agent-config.properties.

  2. Find the following line, and change the value for \(N\) (in minutes):

    agent.sdk_framework.retry.deleteRetryableMessageAfter=N

    Tip

    To keep all messages indefinitely, use the value of -1 for N.

  3. Restart the agent.