Manage and troubleshoot your Integration Agent

There are some basic tasks you can follow and logs you can review to get information on the status of your Integration Agent and specific integrations, and to troubleshoot issues you might come across.

Stopping the Integration Agent

You might need to stop the Integration Agent following integration script or Integration Agent configuration changes, or to clear the inbound queues.

Follow the instructions for your operating system to stop the Integration Agent.

Integration service runtime states

Each integration service running within the Integration Agent has a runtime state. These states are specific to the integration services running within a single Integration Agent (if you have multiple Integration Agents configured.

The states displayed in the xMatters web user interface may be different because they represent the states of an integration service across all Integration Agents providing that service.

IAdmin tool

The Integration Agent includes a command line tool, named IAdmin, that you can use to issue commands to and get the status of the Integration Agent after it has started.

IAdmin is located at:

  • Windows: <IAHOME>\bin\iadmin.bat
  • Linux: <IAHOME>/bin/iadmin.sh

IAdmin Logging

The results of administrative commands in the IAdmin tool are displayed in the console and captured in the log files.

Additionally, exit codes are captured in the log files. If errors occur during the execution of a command, a brief description of the error is displayed on the console, and additional details are captured in the log files.

Integration Agent log

The log files for the Integration Agent and the IAdmin tool are the first place to check if you notice issues with your installation.

Integration Agent logs: Contains logs for all Integration Agent entries of a warning level or higher (by default).

  • Windows: <IAHOME>\log\IntegrationAgent.txt
  • Linux: <IAHOME>/log/IntegrationAgent.txt

IAdmin log: Contains the results of administrative commands in the IAdmin tool.

  • Windows: <IAHOME>\log\IntegrationAgentIAdmin.txt
  • Linux: <IAHOME>/log/IntegrationAgentIAdmin.txt

The Integration Agent uses log4j version 2.17.1 for logging. The information here covers items that are specific to the Integration Agent. See the log4j documentation for full details about its format and its settings.

Default log entry format

Log entries are in the following format:

<date> <time> <thread> <log_level> <log_message>

Logging configuration file

The logging configuration file is named log4j2.xml and is located at:

  • Windows: <IAHOME>\conf
  • Linux: <IAHOME>/conf

To apply logging configuration changes, restart the Integration Agent after saving the configuration file.

Logging categories

Most Integration Agent log messages appear under the com.alarmpoint.integrationagent category hierarchy. Each major component or activity (for example, Health Monitor, heartbeats, integration service requests, etc.) logs to a dedicated subcategory. The advantage to this approach is that logging can be focused on a specific aspect of the Integration Agent’s activities.

To expose a specific logging category, open the log4j2.xml configuration file and un-comment one or more logging categories.

Troubleshooting

If you encounter any hiccups using the Integration Agent, there are some things you can check to help fix the issue.

Startup issues

On startup, the Integration Agent validates its configuration, including the:

  • Presence of the logl4j-bridge.xml configuration file.
  • Presence and well-formedness of the IAConfig.xml file.
  • Availability of the Admin and Integration Agent port.

Not all incorrect configurations prevent startup. For example, problems with the xMatters web server URLs or the integration service configuration files are logged as errors, but do not prevent the Integration Agent from starting. You can inspect the startup exit codes to investigate startup issues.

Unix error code 45

Typically, the Integration Agent is started by a non-privileged user. If the Integration Agent daemon is started using a root account and the hosting machine needs to be restarted for any reason, the service may not start after boot up and return an error code 45. This is caused by the root account owning some of the required folders and denying access to the non-privileged user.

SSL certificates error

When starting up the Integration Agent, you might encounter an error if the SSL certificates were not imported properly.

For more information about SSL troubleshooting for xMatters integrations, visit our support site.

Linux process issues

In some cases, a user may be unable to start the Integration Agent due to an “AlarmPoint_Integration_Agent is already running” exception when the Integration Agent is not actually running. This is because the Integration Agent may not have been stopped properly (e.g., due to a power failure).

To verify that the Integration Agent is not running, search for a process with the argument containing the keyword “wrapper” or “java” that refers to the Integration Agent’s install directory.

Integration service request issues

There are two main types of integration service issues that can occur: integration service request receives no SOAP response and integration service receives SOAP error response.

Integration service request receives No SOAP response

Usually this type of issue indicates one of the following:

  • Improperly installed integration service: For example, caused by a problem with the Integration Agent or integration service configuration
  • Integration service addressing problem: Integration Agent is reachable, but the URL used to specify the Web Service Gateway is invalid (e.g., incorrectly named integration service)
  • Network failure: Integration Agent unreachable from the client

Integration service receives SOAP Error response

This indicates that the integration service is able to receive requests, but not process them.

Heartbeat issues

Usually this type of issue indicates one of the following:

  • Incorrect Integration Agent configuration: For example, the xMatters web server URL is improperly specified.
  • Connectivity issue between xMatters web server and the Integration Agent.

Integration service configuration issues

An integration service with a runtime state of ERROR has a problem with its configuration file. The most likely cause is a syntax error in the integration service's JavaScript.

Notification delay and backlog issues

There's a delay between notification requests being sent to the Integration Agent and delivery of notifications from xMatters. A secondary symptom is a backlog of notification requests in the Integration Agent's inbound queues.

The iadmin get-status command reports a significant number of requests under the integration service's "Normal Priority inbound APXML queue" and/or the "High Priority inbound APXML queue" headings, and these numbers do not decrease substantially when the command is repeated. New notification requests will not reach xMatters until this backlog has been cleared.

Password utility issues

If you are attempting to run the IAPassword utility, and receive a "The system cannot find the specified path" error, install the AdoptOpenJDK archive on your system.