Monitor a Jitterbit private agent with Elasticsearch

Introduction

This page shows how you can remotely monitor the performance and behavior of Docker, Linux, or Windows private agents using the Elasticsearch and Kibana observability platform. Once properly configured, your private agents' metrics can be viewed using the Jitterbit-provided pre-built dashboard, or your own custom dashboard.

This feature is currently released as a beta version, supported on private agents version 11.37.0.7 or later. To provide feedback, contact the Jitterbit Product Team.

To set up Elasticsearch and Kibana, follow the steps in these sections in order:

1. Satisfy the prerequisites.

2. Install the Metricbeat and Filebeat agents on the private agent host.

3. Set the Kibana lifecycle policy.

4. Create Kibana templates.

5. Configure the Metricbeat and Filebeat agents.

6. (Optional) Import a dashboard.

If you have any problems, consult the Troubleshoot section, or contact Jitterbit support.

Prerequisites

To use Elasticsearch and Kibana to monitor a private agent host, you must have the following:

• Private agent 11.37.0.7 or later installed and running.

• A working Elasticsearch and Kibana system installed and configured. To install these, refer to the following pages:

  • Elasticsearch

  • Kibana

Install the Metricbeat and Filebeat agents

Metricbeat and Filebeat are data collecting agents that forward private agent metrics to your Elasticsearch/Kibana system. To install them, follow these steps:

• Metricbeat:

  1. To install Metricbeat on a private agent host, follow the Metricbeat instructions on the Elastic.co website.

  2. To start Metricbeat when the host boots, follow the Metricbeat startup instructions on the Elastic.co website.

  3. To check the Metricbeat connection to Elasticsearch, run the following command:

     metricbeat test output
     

• Filebeat:

  1. To install Filebeat on a private agent host, follow the Filebeat instructions on the Elastic.co website.

  2. To start Filebeat when the host boots, follow the Filebeat startup instructions on the Elastic.co website.

  3. To check the Filebeat connection to Elasticsearch, run the following command:

     filebeat test output
     

Set the Kibana index lifecycle policy

To set the data retention policy, follow these steps:

1. Open your Kibana web console.

2. Enter index lifecycle policies in the search bar, then select the resulting page.

3. Click the Create policy button.

4. In the Create policy dialog, set the following values:

   • Policy name: private-agent-metrics-policy

5. Turn on the Warm phase toggle, then set the following values:

   • Move data into phase when: 30 days old.

6. Turn on the Cold phase toggle, then set the following values:

   • Move data into phase when: 90 days old.

7. Click the Save policy button, then close the summary drawer.

Create Kibana templates

To create templates, follow these steps:

1. Create the private-agent-metric-templatetemplate:

   1. Enter index management in the search bar, then select the resulting page.

   2. Select the Index Templates tab.

   3. Click the Create template button.

   4. On the Logistics page, enter values for the following fields:

      • Name: private-agent-metric-template

      • Index patterns: private-agent-metric-8.15-*

   5. Click the Next button.

   6. On the Component templates page, click the Next button.

   7. On the Index settings page, replace the contents of the Index settings field with the following:

      {
        "index": {
          "lifecycle": {
            "name": "private-agent-metrics-policy",
            "rollover_alias": "private-agent-metric-alias"
          },
          "number_of_shards": "1",
          "number_of_replicas": "1"
        }
      }
      

   8. Click the Next button.

   9. On the Mappings page, in the Mapped fields tab, add fields according to the following table. Click the Add field button after each entry:

      Field type	Field name
      Keyword	fields.env
      Keyword	private-agent.group
      Keyword	private-agent.name

   10. Select the Advanced options tab, then set the following toggles to On:

       • Map numeric strings as numbers

       • Map date strings as dates

   11. Click the Next button.

   12. On the Aliases page, click the Next button.

   13. On the Review details page, click the Create template button, then close the summary drawer.

2. Create the private-agent-filebeat-template template:

   1. On the Index Templates tab, click the Create template button.

   2. On the Logistics page, enter values for the following fields:

      • Name: private-agent-filebeat-template

      • Index patterns: private-agent-filebeat-8.15-*

   3. Click the Next button.

   4. On the Component templates page, click the Next button.

   5. On the Index settings page, replace the contents of the Index settings field with the following:

      {
        "index": {
          "lifecycle": {
            "name": "private-agent-metrics-policy",
            "rollover_alias": "private-agent-metric-alias"
          },
          "number_of_shards": "1",
          "number_of_replicas": "1"
        }
      }
      

   6. Click the Next button.

   7. On the Mappings page, in the Mapped fields tab, add fields according to the following table. Click the Add field button after each entry:

      Field type	Field name
      Keyword	message.fields.environment_name
      Keyword	message.fields.operation_name
      Keyword	message.fields.project_name
      Keyword	message.fields.status
      Keyword	private-agent.group
      Keyword	private-agent.name

   8. Select the Advanced options tab, then set the following toggles to On:

      • Map numeric strings as numbers

      • Map date strings as dates

   9. Click the Next button.

   10. On the Aliases page, click the Next button.

   11. On the Review details page, click the Create template button, then close the summary drawer.

Configure the Metricbeat and Filebeat agents

The Jitterbit private agent includes a script to configure Metricbeat and Filebeat. To use it when installing for the first time, or when upgrading from a private agent 11.34-based installation, follow these steps:

1. Run the following in a terminal or PowerShell:

   DockerLinuxWindows

   cd /opt/jitterbit/scripts/
   ./configure_elasticsearch.sh
   

   cd /opt/jitterbit/scripts/
   ./configure_elasticsearch.sh
   

   cd "C:\Program Files\Jitterbit Agent\scripts\"
   .\configure_elasticsearch.ps
   

2. Read and respond to the following prompts:

   1. Prompt 1:

      *************
      ***WARNING***
      *************
      Installing the metric configurations will backup and overwrite your metricbeat configuration files.
      Do you wish to proceed?
      1) Yes
      2) No
      

      Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:

      • 1: Continue the script. If you have existing Metricbeat and Filebeat configuration files (/etc/metricbeat/metricbeat.yml and /etc/filebeat/filebeat.yml), backup copies are made (/etc/metricbeat/metricbeat.yml.jb.bak and /etc/filebeat/filebeat.yml.jb.bak) and new ones are created.

      • 2: Stop the script.

   2. Prompt 2:

      *************
      ***WARNING***
      *************
      Turning on Agent observability on the agent will turn on Enhanced Capability feature for the agent.
      Enhanced Capability might affect the operation routing behavior for the agents.
      Do you wish to proceed?
      1) Yes
      2) No
      

      Enter the number corresponding to your choice, followed by the Enter key. The choices are as follows:

      • 1: Continue the script. This sets EnhancedCapabilityMetricsEnabled to true in the [AgentCapability] section of jitterbit.conf.

      • 2: Stop the script.

Import an Elasticsearch dashboard

To import a pre-built Elasticsearch dashboard, follow these steps:

1. Download the Jitterbit Private Agent Elasticsearch dashboard JSON file from the Harmony portal Downloads page.

2. Enter kibana saved objects in the Elasticsearch search bar, then select the resulting page.

3. Click the Import button.

4. In the Import saved objects dialog, click Import, find the downloaded dashboard JSON file, then select it.

5. Under Import options, select Check for existing objects with Automatically overwrite conflicts.

6. Click the Import button.

7. Click the Done button.

8. To use the dashboard, enter dashboards in the Elasticsearch search bar, select the resulting page, then select Jitterbit Harmony Private Agent Dashboard.

Troubleshoot Elasticsearch issues

To help solve any issues with Elasticsearch components, you can check the Elasticsearch documentation, inspect log files, or run diagnostic commands.

Elasticsearch documentation

• Metricbeat:

  • Metricbeat FAQ

  • Debug Metricbeat

• Filebeat:

  • Filebeat FAQ

  • Debug Filebeat

Elasticsearch log files locations

• /var/log/metricbeat

• /var/log/filebeat