Observability (Beta) in Jitterbit private agents 11.37 or later

Caution

This feature is currently released as a beta version, supported on private agents version 11.37.0.7 or later. To provide feedback, contact the Jitterbit Product Team.

Introduction

You can remotely monitor the performance and behavior of Docker, Linux, or Windows private agents with any of the following supported observability platforms:

Datadog
Elasticsearch with Kibana

You must install and configure your chosen observability platform on every private agent host you want to monitor. Once properly configured, your agents' metrics can be viewed using the Jitterbit-provided pre-built dashboard, or your own custom dashboard.

Use Datadog to monitor a Jitterbit private agent

Prerequisites

To use Datadog to monitor a private agent host, you must have the following:

Private agent 11.37.0.7 or later installed and running.
A Datadog account.
A super-user (root) account on Linux, or an Administrator account on Windows.

Important

Run all commands as this user.

Install the Datadog agent

To install the Datadog agent on a private agent host, follow these steps:

Log into your Datadog account.
From the main menu, select Integrations > Agent.
On the Agent Installations Instructions page, select your private agent host type.
Click the API Key button. The Select an API Key dialog opens.
If you don't have any API keys, click the Create New button to create one. Otherwise, select your API key entry, then click the Use API Key button.
In the Agent Installation Command, click the Click to copy icon to copy the command to your clipboard.
Open a terminal or PowerShell on the private agent host, paste the copied command, then press Enter to run it.
Close the terminal or PowerShell.

Note

To enable observability in Docker private agents, you must use environment variables.

Configure the Datadog agent

The Jitterbit private agent software includes a script to configure Datadog. To use it when installing for the first time, or when upgrading from a private agent 11.34-based installation, follow these steps:

For Docker private agents, the value for the hostname property in the /etc/datadog-agent/datadog.yaml should be set to the host's hostname. If it is not, set it manually.

Run the following in a terminal or PowerShell on the private agent host:

DockerLinuxWindows

cd /opt/jitterbit/scripts/
./configure_datadog.sh

cd /opt/jitterbit/scripts/
./configure_datadog.sh

cd "C:\Program Files\Jitterbit Agent\scripts\"
.\configure_datadog.ps

Read and respond to the following prompts:
1. Prompt 1:
```
*************
***WARNING***
*************
Installing the metric configurations will backup and overwrite your datadog configuration files.
Do you wish to proceed?
1) Yes
2) No
```
  Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:
  - 1: Continue the script. If you have an existing Datadog configuration file (/etc/datadog-agent/datadog.yaml), a backup copy is made (/etc/datadog-agent/datadog.yml.jb.bak) and a new one is created.
  - 2: Stop the script.
2. Prompt 2:
```
*************
***WARNING***
*************
Turning on Agent observability on the agent will turn on Enhanced Capability feature for the agent.
Enhanced Capability might affect the operation routing behavior for the agents.
Do you wish to proceed?
1) Yes
2) No
```
  Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:
  - 1: Continue the script. This sets EnhancedCapabilityMetricsEnabled to true in the [AgentCapability] section of jitterbit.conf.
  - 2: Stop the script.
3. Prompt 3:
```
Jitterbit agent needs to restart in order to load the new changes.
Do you wish to restart now? This will stop any in progress operations
1) Yes
2) No
```
  Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:
  - 1: Continue to hard stop and restart the private agent, restart the Datadog agent, then exit the script. To confirm your Datadog agent is working, log into your Datadog account, then select Integrations > Fleet Automation from the main menu.
  - 2: Exit the script without restarting the private agent or the Datadog agent. You should restart both the private agent and the Datadog agent when convenient.

Important

On Docker, the Datadog agent does not start automatically with the host. You must manually start it with the following command:

sudo datadog-agent start

Configure Datadog metrics

To define Datadog facets, follow these steps:

Select Logs > Explorer.
Filter logs by your agent's Host, and the JitterbitAgentMetric service.
(Optional) Check that the logs for your agent are in JSON format. To do this, select a recent log entry, then check that the CONTENT column contains a JSON-formatted log message.
Under the Search facets bar, click Add (Add a facet).
In the Add facet dialog's Path field, enter the text shown below, then click the Add button. Repeat for each item in the following list:
- @environment_id
- @environment_name
- @is_operation_over_schedule
- @name
- @operation_id
- @operation_instance_guid
- @operation_name
- @organization_id
- @project_guid
- @project_name
- @status

Create a calculated field

To create a calculated field, follow these steps:

Select Logs > Explorer.
Click the Add button, then select Calculated Field.
In the Create a calculated field dialog, set values for the following fields:
- Name your field: operation_duration_seconds
- Define your formula: @fields.duration_seconds
Click the Save button.

Create a measure

To create a measure, follow these steps:

Select Logs > Explorer.
Under the Search facets bar, click Add (Add a facet).
In the Add facet dialog, select the Measure tab.
In the Path field, enter @operation_duration_seconds.
Click the Add button.

Create operation metrics

To define operation metrics, select Logs > Generate Metrics, then follow the steps below for each operation metric.

Tip

You can also use logs to create Datadog metrics.

Create the `metric.operation.count.by.status` operation metric

Click the New Metric button.
In the Generate Metric dialog, set values as follows:
- Set Metric Name: metric.operation.count.by.status
- Define Query: service:JitterbitAgentMetric @name:operation_log
Click the group by menu, then click each of the following to add them to the list:
- @fields.operation_id
- @fields.operation_name
- @fields.status
- @agentgroup
- @host
Click the Create Metric button.

Create the `metric.operation.running.over.scheduled.interval` operation metric

Click the New Metric button.
In the Generate Metric dialog, set values as follows:
- Set Metric Name: metric.operation.running.over.scheduled.interval
- Define Query: service:JitterbitAgentMetric @name:operation_running_over_scheduled_interval
Click the group by menu, and click each of the following to add them to the list:
- @fields.operation_id
- @fields.operation_name
- @agentgroup
- @host
Click the Create Metric button.

Create the `metric.operation.duration.seconds` operation metric

Click the New Metric button.
In the Generate Metric dialog, set values as follows:
- Set Metric Name: metric.operation.duration.seconds
- Define Query: service:JitterbitAgentMetric @name:operation_running_over_scheduled_interval
Click the group by menu, and click each of the following to add them to the list:
- @fields.operation_id
- @fields.operation_name
- @agentgroup
- @host
Click the Create Metric button.

Create process metrics

To define process metrics, select Infrastructure > Processes, select the Manage Metrics tab, then follow the steps below for each process metric.

Create the `proc.openginebyname.cpu.num_threads` process metric

Click the New Metric button.
Under section 1, Filter Processes, in the Filter by field, enter command:JitterbitOperationEngineProc.
Under section 2, Select Measure and Dimensions, set values for the following fields:
- Open the Measure menu, then select # of Threads.
- Open the Dimensions menu, then select name.
Under section 4, Name, in the metric.name field, enter openginebyname to make the name proc.openginebyname.cpu.num_threads.
Click the Create button.

Create the `proc.operationsengine.cpu.num_threads` process metric

Click the New Metric button.
Under section 1, Filter Processes, in the Filter by field, enter command:JitterbitOperationEngineProc.
Under section 2, Select Measure and Dimensions, set values for the following fields:
- Open the Measure menu, then select # of Threads.
- Open the Dimensions menu, then select agentgroup.
Under section 4, Name, in the metric.name field, enter operationsengine to make the name proc.operationsengine.cpu.num_threads.
Click the Create button.

Import a Datadog dashboard

To import Jitterbit's pre-built Datadog dashboard, follow these steps:

Download the Jitterbit Private Agent Datadog dashboard JSON file from the Harmony portal Downloads page.
Select Dashboards > New Dashboard.
In the Create a Dashboard dialog, enter a name in the Dashboard Name field, then click the New Dashboard button.
Click the Configure button, then select Import dashboard JSON.
Find the downloaded dashboard JSON file, then select it.
To use the dashboard, select Dashboards > Dashboard list, enter Jitterbit Harmony Private Agent in the Search dashboards field, then select the imported dashboard.

Troubleshoot Datadog issues

To help solve any issues with Datadog, you can check the Datadog documentation, inspect log files, or run commands to manage users and permissions.

Datadog documentation

Datadog file locations

Log files

Datadog log files can be found in the following location:

LinuxWindows

/var/log/datadog/

Open the Datadog Agent Manager application, then select the Log tab.

Configuration files

The Datadog configuration file can be found in the following location:

LinuxWindows

/etc/datadog-agent/datadog.yaml

Open the Datadog Agent Manager application, then select the Settings tab.

In this file, you should check that you have correct values for the following keys:

api_key
site
$hostname
tags

Datadog users and permissions commands

To create a Datadog user and group, run these commands:

usermod -a -G root dd-agent
usermod -a -G jitterbit dd-agent

To set Datadog configuration file ownership, run these commands:

chown dd-agent:dd-agent /etc/datadog-agent/conf.d/logs.d/conf.yaml
chown dd-agent:dd-agent /etc/datadog-agent/conf.d/logs.d

Use Elasticsearch and Kibana to monitor a Jitterbit private agent

Prerequisites

To use Elasticsearch and Kibana to monitor a private agent host, you must have the following:

Private agent 11.37.0.7 or later installed and running.
A working Elasticsearch and Kibana system installed and configured. To install these, refer to the following pages:
- Elasticsearch
- Kibana

Install Metricbeat and Filebeat

Metricbeat and Filebeat are data collecting agents that forward private agent metrics to your Elasticsearch/Kibana system. To install them, follow these steps:

Metricbeat:
1. To install Metricbeat on a private agent host, follow the Metricbeat instructions on the Elastic.co website.
2. To start Metricbeat when the host boots, follow the Metricbeat startup instructions on the Elastic.co website.
3. To check the Metricbeat connection to Elasticsearch, run the following command:
```
metricbeat test output
```
Filebeat:
1. To install Filebeat on a private agent host, follow the Filebeat instructions on the Elastic.co website.
2. To start Filebeat when the host boots, follow the Filebeat startup instructions on the Elastic.co website.
3. To check the Filebeat connection to Elasticsearch, run the following command:
```
filebeat test output
```

Set the Kibana index lifecycle policy

To set the data retention policy, follow these steps:

Open your Kibana web console.
Enter index lifecycle policies in the search bar, then select the resulting page.
Click the Create policy button.
In the Create policy dialog, set the following values:
- Policy name: private-agent-metrics-policy
Turn on the Warm phase toggle, then set the following values:
- Move data into phase when: 30 days old.
Turn on the Cold phase toggle, then set the following values:
- Move data into phase when: 90 days old.
Click the Save policy button, then close the summary drawer.

Create Kibana templates

To create templates, follow these steps:

Create the private-agent-metric-templatetemplate:
1. Enter index management in the search bar, then select the resulting page.
2. Select the Index Templates tab.
3. Click the Create template button.
4. On the Logistics page, enter values for the following fields:
  - Name: private-agent-metric-template
  - Index patterns: private-agent-metric-8.15-*
5. Click the Next button.
6. On the Component templates page, click the Next button.
7. On the Index settings page, replace the contents of the Index settings field with the following:
```
{
  "index": {
    "lifecycle": {
      "name": "private-agent-metrics-policy",
      "rollover_alias": "private-agent-metric-alias"
    },
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }
}
```
8. Click the Next button.
9. On the Mappings page, in the Mapped fields tab, add fields according to the following table. Click the Add field button after each entry:
  
  Field type Field name
  
  Keyword fields.env
  
  Keyword private-agent.group
  
  Keyword private-agent.name
10. Select the Advanced options tab, then set the following toggles to On:
  - Map numeric strings as numbers
  - Map date strings as dates
11. Click the Next button.
12. On the Aliases page, click the Next button.
13. On the Review details page, click the Create template button, then close the summary drawer.

Create the private-agent-filebeat-template template:

On the Index Templates tab, click the Create template button.
On the Logistics page, enter values for the following fields:
- Name: private-agent-filebeat-template
- Index patterns: private-agent-filebeat-8.15-*
Click the Next button.
On the Component templates page, click the Next button.

On the Index settings page, replace the contents of the Index settings field with the following:

{
  "index": {
    "lifecycle": {
      "name": "private-agent-metrics-policy",
      "rollover_alias": "private-agent-metric-alias"
    },
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }
}

Click the Next button.

On the Mappings page, in the Mapped fields tab, add fields according to the following table. Click the Add field button after each entry:

Field type	Field name
Keyword	`message.fields.environment_name`
Keyword	`message.fields.operation_name`
Keyword	`message.fields.project_name`
Keyword	`message.fields.status`
Keyword	`private-agent.group`
Keyword	`private-agent.name`

Select the Advanced options tab, then set the following toggles to On:
- Map numeric strings as numbers
- Map date strings as dates
Click the Next button.
On the Aliases page, click the Next button.
On the Review details page, click the Create template button, then close the summary drawer.

Configure Metricbeat and Filebeat

The Jitterbit private agent includes a script to configure Metricbeat and Filebeat. To use it when installing for the first time, or when upgrading from a private agent 11.34-based installation, follow these steps:

Run the following in a terminal or PowerShell:

DockerLinuxWindows

cd /opt/jitterbit/scripts/
./configure_elasticsearch.sh

cd /opt/jitterbit/scripts/
./configure_elasticsearch.sh

cd "C:\Program Files\Jitterbit Agent\scripts\"
.\configure_elasticsearch.ps

Read and respond to the following prompts:
1. Prompt 1:
```
*************
***WARNING***
*************
Installing the metric configurations will backup and overwrite your metricbeat configuration files.
Do you wish to proceed?
1) Yes
2) No
```
  Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:
  - 1: Continue the script. If you have existing Metricbeat and Filebeat configuration files (/etc/metricbeat/metricbeat.yml and /etc/filebeat/filebeat.yml), backup copies are made (/etc/metricbeat/metricbeat.yml.jb.bak and /etc/filebeat/filebeat.yml.jb.bak) and new ones are created.
  - 2: Stop the script.
2. Prompt 2:
```
*************
***WARNING***
*************
Turning on Agent observability on the agent will turn on Enhanced Capability feature for the agent.
Enhanced Capability might affect the operation routing behavior for the agents.
Do you wish to proceed?
1) Yes
2) No
```
  Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:
  - 1: Continue the script. This sets EnhancedCapabilityMetricsEnabled to true in the [AgentCapability] section of jitterbit.conf.
  - 2: Stop the script.

Note

To enable observability in Docker private agents, you must use environment variables.

Import an Elasticsearch dashboard

To import a pre-built Elasticsearch dashboard, follow these steps:

Download the Jitterbit Private Agent Elasticsearch dashboard JSON file from the Harmony portal Downloads page.
Enter kibana saved objects in the Elasticsearch search bar, then select the resulting page.
Click the Import button.
In the Import saved objects dialog, click Import, find the downloaded dashboard JSON file, then select it.
Under Import options, select Check for existing objects with Automatically overwrite conflicts.
Click the Import button.
Click the Done button.
To use the dashboard, enter dashboards in the Elasticsearch search bar, select the resulting page, then select Jitterbit Harmony Private Agent Dashboard.

Troubleshoot Elasticsearch issues

To help solve any issues with Elasticsearch components, you can check the Elasticsearch documentation, inspect log files, or run diagnostic commands.

Elasticsearch documentation

Metricbeat:
- Metricbeat FAQ
- Debug Metricbeat
Filebeat:
- Filebeat FAQ
- Debug Filebeat

Elasticsearch log files locations

/var/log/metricbeat
/var/log/filebeat

Field type	Field name
Keyword	`fields.env`
Keyword	`private-agent.group`
Keyword	`private-agent.name`

Observability (Beta) in Jitterbit private agents 11.37 or later

Introduction

Use Datadog to monitor a Jitterbit private agent

Prerequisites

Install the Datadog agent

Configure the Datadog agent

Configure Datadog metrics

Create facets

Create a calculated field

Create a measure

Create operation metrics

Create the metric.operation.count.by.status operation metric

Create the metric.operation.running.over.scheduled.interval operation metric

Create the metric.operation.duration.seconds operation metric

Create process metrics

Create the proc.openginebyname.cpu.num_threads process metric

Create the proc.operationsengine.cpu.num_threads process metric

Import a Datadog dashboard

Troubleshoot Datadog issues

Datadog documentation

Datadog file locations

Log files

Configuration files

Datadog users and permissions commands

Use Elasticsearch and Kibana to monitor a Jitterbit private agent

Prerequisites

Install Metricbeat and Filebeat

Set the Kibana index lifecycle policy

Create Kibana templates

Configure Metricbeat and Filebeat

Import an Elasticsearch dashboard

Troubleshoot Elasticsearch issues

Elasticsearch documentation

Elasticsearch log files locations

Create the `metric.operation.count.by.status` operation metric

Create the `metric.operation.running.over.scheduled.interval` operation metric

Create the `metric.operation.duration.seconds` operation metric

Create the `proc.openginebyname.cpu.num_threads` process metric

Create the `proc.operationsengine.cpu.num_threads` process metric