Skip to Content

Monitor a Jitterbit private agent with Datadog

Introduction

This page shows how you can remotely monitor the performance and behavior of Docker, Linux, or Windows private agents using the Datadog observability platform. Once properly configured, your private agents' metrics can be viewed using the Jitterbit-provided pre-built dashboard, or your own custom dashboard.

This feature is currently released as a beta version, supported on private agents version 11.37.0.7 or later. To provide feedback, contact the Jitterbit Product Team.

To set up Datadog, follow the steps in these sections in order:

  1. Satisfy the prerequisites.

  2. Install the Datadog agent on the private agent host.

  3. Configure the Datadog agent.

  4. Configure Datadog metrics.

  5. (Optional) Import a Datadog dashboard.

If you have any problems, consult the Troubleshoot section, or contact Jitterbit support.

Prerequisites

You must have the following:

  • Private agent 11.37.0.7 or later installed and running.

  • A Datadog account.

  • A super-user (root) account on Linux, or an Administrator account on Windows.

    Important

    Run all commands on this page as this user.

  • A valid RFC 1123 hostname for the host where the private agent is running.

  • Additional requirements for Docker private agents:

Install the Datadog agent

To install the Datadog agent on a private agent host, follow these steps:

  1. Log into your Datadog account.

  2. From the main menu, select Integrations > Fleet Automation > Install Agents.

  3. On the Select platform to install Agents page, select the platform that your private agent is running on:

    Select Host based > Linux.

    Select Host based > Linux.

    Select Host based > Windows.

  4. On the Install the Datadog Agent on page, there are these three sections:

    • ① Customize your observability coverage: Enable required features, and select an environment tag.

    • ② Run the install command:

      1. Click the Select an API Key button. The Select an API Key dialog opens.

      2. Select an API key from the list, then click the Use API Key button.

        If you don't have a key. click the Create New button to create one.

      3. Open a console (terminal or PowerShell) on the private agent host. Copy and paste the code block into the console.

      4. Do the following:

        1. Add a line to set the value for the DD_HOSTNAME environment variable to the private agent's hostname.

        2. Change install_script_agent7.sh to install_script_agent6.sh.

          (Currently, the Jitterbit private agent only supports Datadog agent version 6.)

        3. Your command should like like this, with your own API key (line 1), private agent hostname (line 3), and edited curl argument (line 4):

          1
2
3
4
          DD_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
DD_SITE="datadoghq.com" \
DD_HOSTNAME="hostname" \
bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent6.sh)"

        4. Press Enter to run the commands.

        Press Enter to run the commands.

        Press Enter to run the commands.

    • ③ Confirm the Agent installation: When you have the Datadog agent installed and running on your private agent host, wait a while for it to appear in this panel before continuing.

  5. (Docker only) Create a default Datadog agent security file:

    After installing, the Datadog agent fails to start due to a known Datadog issue. To correct this, make a copy of the example security agent file before proceeding to the next section:

    cp  /etc/datadog-agent/security-agent.yaml.example /etc/datadog-agent/security-agent.yaml

    No action needed.

    No action needed.

Configure the Datadog agent

The Jitterbit private agent software includes a script to configure and start the Datadog agent. You can use it when installing for the first time, or when upgrading from a private agent 11.34-based installation. To do so, follow these steps:

  1. Run the following in a terminal or PowerShell on the private agent host:

    cd /opt/jitterbit/scripts/
./configure_datadog.sh

    cd /opt/jitterbit/scripts/
./configure_datadog.sh

    cd "C:\Program Files\Jitterbit Agent\scripts\"
.\configure_datadog.ps

  2. Read and respond to the following prompts:

    1. Prompt 1:

      *************
***WARNING***
*************
Installing the metric configurations will backup and overwrite your datadog configuration files.
Do you wish to proceed?
1) Yes
2) No

      Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:

      • 1: Continue the script. If you have an existing Datadog configuration file (/etc/datadog-agent/datadog.yaml), a backup copy is made (/etc/datadog-agent/datadog.yml.jb.bak) and a new one is created.

      • 2: Stop the script.

    2. Prompt 2:

      *************
***WARNING***
*************
Turning on Agent observability on the agent will turn on Enhanced Capability feature for the agent.
Enhanced Capability might affect the operation routing behavior for the agents.
Do you wish to proceed?
1) Yes
2) No

      Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:

    3. Prompt 3:

      Jitterbit agent needs to restart in order to load the new changes.
Do you wish to restart now? This will stop any in progress operations
1) Yes
2) No

      Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:

      • 1: Continue to hard stop and restart the private agent, restart the Datadog agent, then exit the script. To confirm your Datadog agent is working, log into your Datadog account, then select Integrations > Fleet Automation from the main menu.

      • 2: Exit the script without restarting the private agent or the Datadog agent. You should restart both the private agent and the Datadog agent when convenient.

Important

On Docker, the Datadog agent does not start automatically with the host. You must manually start it with the following command:

sudo datadog-agent run

Configure Datadog metrics

Create facets

To define Datadog facets, follow these steps:

  1. Select Logs > Explorer.

  2. In the Search for bar, filter logs by your agent's Host, and the JitterbitAgentMetric service using the following string:

    host:<hostname> service:JitterbitAgentMetric

  3. (Optional) Check that the logs for your agent are in JSON format. To do this, select a recent log entry, then check that the CONTENT column contains a JSON-formatted log message.

  4. Under the Search facets bar, click the Add button (Add a facet). The Add facet dialog opens.

  5. In the Path field, enter the text shown below, then click the Add button. Repeat for each item in the following list:

    • @environment_id

    • @environment_name

    • @is_operation_over_schedule

    • @name

    • @operation_id

    • @operation_instance_guid

    • @operation_name

    • @organization_id

    • @project_guid

    • @project_name

    • @status

Create a calculated field

To create a calculated field, follow these steps:

  1. Select Logs > Explorer.

  2. Click the Add button, then select Calculated Field. The Create a calculated field dialog opens.

  3. Set values for the following fields:

    • Name your field: operation_duration_seconds

    • Define your formula: @fields.duration_seconds

  4. Click the Save button.

Create a measure

To create a measure, follow these steps:

  1. Select Logs > Explorer.

  2. Under the Search facets bar, click the Add button (Add a facet). The Add facet dialog opens.

  3. Select the Measure tab.

  4. In the Path field, enter @operation_duration_seconds.

  5. Click the Add button.

Create operation metrics

To define operation metrics, select Logs > Generate Metrics, then follow the steps below for each operation metric.

Tip

You can also use logs to create Datadog metrics.

Create the metric.operation.count.by.status operation metric

  1. Click the New Metric button.

  2. In the Generate Metric dialog, set values as follows:

    • Set Metric Name: metric.operation.count.by.status

    • Define Query: service:JitterbitAgentMetric @name:operation_log

  3. Click the group by menu, then click each of the following to add them to the list:

    • @fields.operation_id

    • @fields.operation_name

    • @fields.status

    • @agentgroup

    • @host

  4. Click the Create Metric button.

Create the metric.operation.running.over.scheduled.interval operation metric

  1. Click the New Metric button.

  2. In the Generate Metric dialog, set values as follows:

    • Set Metric Name: metric.operation.running.over.scheduled.interval

    • Define Query: service:JitterbitAgentMetric @name:operation_running_over_scheduled_interval

  3. Click the group by menu, and click each of the following to add them to the list:

    • @fields.operation_id

    • @fields.operation_name

    • @agentgroup

    • @host

  4. Click the Create Metric button.

Create the metric.operation.duration.seconds operation metric

  1. Click the New Metric button.

  2. In the Generate Metric dialog, set values as follows:

    • Set Metric Name: metric.operation.duration.seconds

    • Define Query: service:JitterbitAgentMetric @name:operation_running_over_scheduled_interval

  3. Click the group by menu, and click each of the following to add them to the list:

    • @fields.operation_id

    • @fields.operation_name

    • @agentgroup

    • @host

  4. Click the Create Metric button.

Create process metrics

To define process metrics, select Infrastructure > Processes, select the Manage Metrics tab, then follow the steps below for each process metric.

Create the proc.openginebyname.cpu.num_threads process metric

  1. Click the New Metric button.

  2. Under section 1, Filter Processes, in the Filter by field, enter command:JitterbitOperationEngineProc.

  3. Under section 2, Select Measure and Dimensions, set values for the following fields:

    • Open the Measure menu, then select # of Threads.

    • Open the Dimensions menu, then select name.

  4. Under section 4, Name, in the metric.name field, enter openginebyname to make the name proc.openginebyname.cpu.num_threads.

  5. Click the Create button.

Create the proc.operationsengine.cpu.num_threads process metric

  1. Click the New Metric button.

  2. Under section 1, Filter Processes, in the Filter by field, enter command:JitterbitOperationEngineProc.

  3. Under section 2, Select Measure and Dimensions, set values for the following fields:

    • Open the Measure menu, then select # of Threads.

    • Open the Dimensions menu, then select agentgroup.

  4. Under section 4, Name, in the metric.name field, enter operationsengine to make the name proc.operationsengine.cpu.num_threads.

  5. Click the Create button.

Import a Datadog dashboard

To import Jitterbit's pre-built Datadog dashboard, follow these steps:

  1. Download the Jitterbit Private Agent Datadog dashboard JSON file from the Harmony portal Downloads page.

  2. Select Dashboards > New Dashboard.

  3. In the Create a Dashboard dialog, enter a name in the Dashboard Name field, then click the New Dashboard button.

  4. Click the Configure button, then select Import dashboard JSON.

  5. Find the downloaded dashboard JSON file, then select it.

  6. To use the dashboard, select Dashboards > Dashboard list, enter Jitterbit Harmony Private Agent in the Search dashboards field, then select the imported dashboard.

Troubleshoot Datadog issues

To help solve any issues with Datadog, you can check the Datadog documentation, inspect log files, or run commands to manage users and permissions.

Datadog documentation

Datadog file locations

Log files

Datadog log files can be found in the following location:

/var/log/datadog/

/var/log/datadog/

Open the Datadog Agent Manager application, then select the Log tab.

Configuration files

The Datadog configuration file can be found in the following location:

/etc/datadog-agent/datadog.yaml

/etc/datadog-agent/datadog.yaml

Open the Datadog Agent Manager application, then select the Settings tab.

In this file, you should check that you have correct values for the following keys:

  • api_key

  • site

  • $hostname

  • tags

Datadog users and permissions commands

To create a Datadog user and group, run these commands:

usermod -a -G root dd-agent
usermod -a -G jitterbit dd-agent

To set Datadog configuration file ownership, run these commands:

chown dd-agent:dd-agent /etc/datadog-agent/conf.d/logs.d/conf.yaml
chown dd-agent:dd-agent /etc/datadog-agent/conf.d/logs.d