Manage and troubleshoot your Integration Agent

To stop the Integration Agent running as a Windows Service, do one of the following:

• Open Windows Administrative Tools > Component Services, right-click xMatters Integration Agent, and then click Stop.
• Run stop_service.bat (located at: <IAHOME>\bin) from a command line.
• Run shutdown.bat (located at: <IAHOME>\bin) from a command line.
  • This commands suspends the agent, waits 30 seconds if there are pending requests, terminates any requests, and then stops the service.

To stop the Integration Agent running as a Windows program:

• Press Ctrl+C in the console window running the agent.

To stop the Integration Agent daemon, do one of the following:

• From a command line, run ./<IAHOME>/bin/stop_daemon.sh
• From a command line, run ./<IAHOME>/bin/shutdown.sh 
  • This commands suspends the agent, waits 30 seconds if there are pending requests, terminates any requests, and then stops the service.

To stop the Integration Agent running as a Linux application:

• Press Ctrl+C in the console window running the agent.

The IAdmin tool's suspend or resume commands changes the service's runtime state between SUSPENDED and ACTIVE.

There are various commands you can use with the iadmin command line tool to manage the Integration Agent and get information on its status.

The IAdmin command line tool returns an exit code number (meant to be used in automation scripts) indicating either success or the type of error encountered.

IAdmin tool exit codes

The following represents a log entry in the default format:

2018/04/25 15:52:01.537 -0700 PDT [applications|sample-plan-1] INFO - Calling JavaScript method apia_http

The following table summarizes selected log4j categories:

By default, the Heartbeat logging category is inactive because it is commented out in the log4j2.xml file:

<!-- 

<logger name="com.alarmpoint.integrationagent.heartbeat">

<level value="DEBUG"/>

</logger>

-->

To enable Heartbeat logging, uncomment the entry and save the file:

<logger name="com.alarmpoint.integrationagent.heartbeat">

<level value="DEBUG"/>

</logger>

• Windows: The service state is “Starting” during this initial validation, and does not change to “Started” until the validation is complete. If the agent cannot start, the reason for the failure is written to the Windows system log and an exit code is set on the xMattersIntegration Agent Service (if the agent was started as a console application, the agent’s exit code is set as the batch file’s exit code).
  • To see the Window’s service exit code, run the following from a command line:
    sc query apia
    The code appears in the SERVICE_EXIT_CODE field.
  • Alternatively, if the shutdown.bat script is used to start the Integration Agent, the batch file returns the service exit code.
    
• Linux: The initial validation is part of the daemon process. Any problems preventing startup cause the daemon to terminate. Details regarding startup problems are sent to the syslog daemon through the user facility at the fatal level and with the identity apia. The syslog daemon must be configured by an administrator to perform logging.

Additionally, on both Windows and Linux, all startup logging is written to the console and a special log file named apia.txt located in the log directory. After startup is complete, logging reverts to the standard Integration Agent log files.

1. Log in as the root account and navigate to the <IAHOME> folder.
2. Run the following commands:

   chown -Rf xm:xm .mule

   chown -Rf xm:xm .activemq-data

   chown -Rf xm:xm log/IntegrationAgent.txt

3. Log out of the root account.
4. Restart the Integration Agent daemon as a non-privileged user.

The error message is similar to the following:

[Heartbeat-1] ERROR - The xMatters Web Server

https://<HOST>.xmatters.com/api/services/AlarmPointWebService is unavailable or completely rejected the heartbeat.

org.apache.axis2.AxisFault: sun.security.validator.ValidatorException:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable

to find valid certification path to requested target

To resolve this error:

1. Stop the Integration Agent.
2. Ensure that the required SSL certificates have been imported into <IAHOME>\jre\lib\security\cacerts.
   • For additional instructions or troubleshooting tips, contact Support.
3. Restart the Integration Agent.

On some Linux systems, your can search for this process by reviewing the output of the following commands:

ps -ef | grep wrapper

ps -ef | grep java

If such processes exist, then the Integration Agent is running and must be stopped before it is restarted.

If no such process exists, then deleting the file located at <IAHOME>/lib/mule-1.4.3/bin/.apia.pid should allow the Integration Agent to start.

1. Make sure the Integration Agent has started then issue a get-status command using IAdmin.
2. Verify that the integration service appears in the list of services and is ACTIVE.
   • If the service does not appear in the list or has an ERROR_ACTIVE or ERROR_INACTIVE status, check the Integration Agent log to determine the nature of the configuration problem.
3. Attempt to reach an integration service by opening a web browser and typing the URL of the service followed by ?wsdl in the address bar.

For example, if an integration service has the following configuration:

• Name: sample-plan
• Event Domain: applications
• IAConfig.xml: <service-gateway ssl="true" host="www.company.com" port="8081"/>

To test this service, type the following into a web browser’s address bar (preferably from a computer in the same location as the client):

https://www.company.com:8081/applications_sample-plan?wsdl

If the integration service is properly installed and accessible, then a response similar to the following is returned:

<?xml version="1.0" encoding="UTF-8" ?> <wsdl:definitions target...<snip>

If this response is received, the problem is likely related to an incorrectly configured Integration Agent client. If this response is not received, the problem is likely related to a connectivity issue (e.g., connection prevented by a firewall) between the Integration Agent client and the Integration Agent.

1. Make sure the Integration Agent has started then issue a get-status command using IAdmin.
2. Verify that the integration service appears in the list of services and is ACTIVE.
   • If the service is in any other state, then the expected behavior is for the service to deny the request and respond with a SOAP error indicating that the service is not able to process the request.
3. In the Integration Agent’s log configuration file, set (or add) a DEBUG category for the specific integration service. For example, for an integration service named sample-plan, the logl4j-bridge.xml entry would be similar to the following:

<logger name="com.alarmpoint.integrationagent.services.applications.sample-plan">

<level value="DEBUG"/>

</logger>

4. Save the log configuration file and wait at least 10 seconds to allow the change to be detected.
5. Submit a new integration service request, and then review the log file to determine the nature of the problem.

1. Make sure the Integration Agent has started then issue a display-settings command using IAdmin.
2. Verify that the xMatters web server URL appears in the list of server URLs.
   • If the URL does not appear in the list, then this indicates a problem in the IAConfig.xml file (e.g., the xMatters web server’s URL is malformed or not specified).
3. In the Integration Agent’s log configuration file, set the heartbeat category to DEBUG; for example:

<logger name="com.alarmpoint.integrationagent.heartbeat">

<level value="DEBUG"/>

</logger>

4. Restart the Integration Agent and wait for an attempt to send a heartbeat to the xMatters web server that is rejecting the heartbeats.
5. Consult the Integration Agent’s log file and locate the ERROR entry associated with the heartbeat attempt. The ERROR message either indicates a connectivity failure or provides one of the following reasons for the heartbeat’s rejection:
   • UNKNOWN_DOMAIN: indicates that the integration service's event domain has not been configured on the xMatters web server
   • UNKNOWN_SERVICE: indicates that one of the integration service names has not been configured on the xMatters web server
   • REGISTRATION_ACL_FAILED: indicates that this Integration Agent’s ID has not been configured on the xMatters web server
   • UNKNOWN_APPLICATION_ERROR: indicates that an unexpected error occurred
   • SERVICE_DENIED: Depending on the error message, the web services user account in xMatters does not have the "Receive APXML" and "Send APXML" permissions (error message contains reference to "ReceiveAPXML") or does not have the "Register Integration Agent" permission (error message references a rejected heartbeat/registration).

1. Review the Integration Agent log file.
2. Locate entries pertaining to parsing the Integration Agent configuration.
3. Within these entries, locate log entries that refer to parsing the integration service configuration file causing the issue. One of these entries contains an error and stack trace identifying the problem.

Example

The following log excerpt shows the context in which integration service configuration errors appear within the Integration Agent log file (the excerpt has been edited for brevity).

The first three log entries show that the integration service configuration is being parsed. The final entry shows that the integration service’s JavaScript contains a syntax error.

2007-11-06 17:28:59,567 [WrapperSimpleAppMain] INFO - Starting to initialize Integration Agent using IA Config file

C:\sandbox\INTEGR~1\distr\INSTAL~1\conf\IAConfig.xml.

2007-11-06 17:28:59,598 [WrapperSimpleAppMain] INFO - Starting to parse the Integration Service Config files in directory ../integrationservices.

2007-11-06 17:28:59,598 [WrapperSimpleAppMain] INFO - Parsing Integration Service Config file netcool/netcool-admin.xml.

2007-11-06 17:28:59,614 [WrapperSimpleAppMain] ERROR - The script for Integration Service (default,test_service_1) could not be created due to an exception.

ScriptCreationException: The script for Integration Service (default,test_service_1) could not be created due to an exception.

Caused by: org.mozilla.javascript.EvaluatorException: missing } after function body

(C:\sandbox\integrationagent\distr\installation\integrationservices\netcool\netcool-admin.js#71)

1. Increase the number of messages logged by the Integration Agent's queue manager and by the integration scripts, so that the cause of the delay can be identified:
   1. In <AHome>/conf/logl4j-bridge.xml, un-comment the "com.alarmpoint.integrationagent.jms" logger and ensure it is set to DEBUG.
   2. Ensure that the "com.alarmpoint.integrationagent.services" logger is un-commented and set to DEBUG.
   3. In the "txtAppender" section, ensure that the "maxBackupIndex" value is 10 and the "MaxFileSize" value is 10000KB.
   4. Save the file.
2. Purge the Integration Agent queue, to allow new notification requests to be processed without delay:
   1. Stop the Integration Agent (<IAHOME>\bin\shutdown).
   2. Delete the .mule and .activemq-data folders from <IAHome>.
   3. Start the Integration Agent.
   4. Inspect the messages in the Integration Agent log file (<IAHome>\log\IntegrationAgent.txt) to verify that new notification requests are processed by the Integration Agent without timeouts or other errors. Use the xMatters web user interface to check that xMatters alerts are created without undue delay.
   5. Use the iadmin get-status command to monitor the queue sizes.

If delays are still observed or queue sizes increase and do not drop again, use the support-zip tool to create an archive and contact Customer Support:

1. Open a command prompt or terminal window.
2. Navigate to cd <IAHome>/tools.
3. Type or paste the command: support-zip. The tool will tell you the name and location of the archive that it creates.
4. Open a support ticket and attach the support-zip archive.