Datadog
Datadog is a cloud monitoring solution that brings metrics from all of your apps, tools, and services into one place.
The integration uses custom webhooks in Datadog to automatically identify and notify the on-call resource on a variety of devices. The integration also allows users to mute and claim alerts from Datadog Monitors or to view xMatters shift data directly from their devices.
Do more with this integration
Incorporate this integration into your orchestrated incident resolution workflows with Flow Designer. After you create a configuration, the Flows tab appears. From there, you can build your flows — add new response options, connect to other apps, or even create new xMatters events based on activities in the flow — enriching the information injected by this integration along the way.
Get started with Datadog

To install this integration in xMatters, simply go to Workflow Templates on the Workflows tab and start typing the name of the integration you're looking for to filter the list. Once you find it, click its tile then click Next to get to the configuration screen.
How to set up a Datadog configuration
After you give your configuration a name and description, type a name to use when sending alerts (so you can easily tell which configuration or integration the alerts are from), and add all of the users and groups you want to be notified.
Next, retrieve your API key, and create an application key for xMatters in Datadog.

- Log into Datadog.
- Click Integrations > APIs.
- You can find your API Key in the Key column of the API Keys list.
- To create a new Application Key for xMatters, type a name for the key (for example, "xMatters") in the New application key field, and click Create Application Key.
- You can now copy the application key from the Hash column in the Application Keys list.
Enter the Datadog API key and the xMatters application key in the fields on the configuration page in xMatters, and then click Save.
Once you save the configuration, xMatters displays the URLs and payload you need to configure Datadog:
Add each of the mappings to the Webhooks configuration in Datadog.

- Log in as an administrator, and then click Integrations > Integrations.
- Locate and click the Webhooks badge.
- On the Configuration tab, click New to create a new webhook.
- In the Name field, type xmatters_alert, then paste the URL for that webhook in the URL field.
- Paste the payload from the integration configuration page in xMatters into the Payload field.
- Click Save.
- Repeat steps 3-6 for the remaining webhooks: xmatters_on_call, xmatters_engage, and xmatters_conference.
- Click Save.
Your integration is now complete!
Use your integration
The integration uses custom webhooks in Datadog that can be @-mentioned in event or monitor alert messages, or even directly in a status update within the event stream. Each webhook sends the same payload to a different xMatters integration, which process the payload to create events in xMatters, or send comments back to the original Datadog event.
You can test the Datadog Alerts feature by notifying the xmatters_alert custom webhook when one of your monitors moves into an Alert or Error state.

- Click Monitors > Manage Monitors.
- Click the monitor with alerts you want to send to xMatters, and then click the Edit tab.
- In the Say what's happening section, type the message you want to send, and include the following to initiate the webhook:
@webhook-xmatters_alert
Notes
- Monitor bodies allow you to send different information depending on the transition of the monitor; for example, when transitioning to alert state or recovery state.
- If you do not want to receive alerts when the monitor moves to a "No_Data" state, you may need to wrap the webhook @-mention in is_alert tags; i.e., {{#is_alert}} {{/is_alert}}
- If you want the integration to automatically terminate xMatters events for recovered alerts, you will need to also wrap a webhook @-mention in {{#is_recovery}} {{/is_recovery}}
- For more details on message template variables, consult the Datadog documentation.

The Engage with xMatters feature allows users to provide updates, request help, or start a conference bridge with colleagues from other teams to collaborate on events, event stream posts or snapshots within Datadog. You can try out the Engage with xMatters feature using the Event Stream or a Snapshot Comment.
To engage with xMatters:
- Click Events.
- In the Leave a status update field for an event, use the following format to initiate the webhook (see the Notes below for more syntax information):
<message_text> @webhook-xmatters_engage <recipient>
- Click Post to create the post in Datadog and trigger the event in xMatters.
- If enabled, outbound integrations for status updates, delivery notifications and response updates will create comments and other updates on the Datadog post.
Notes
Engage with xMatters posts have some important required formatting:
- All text preceding the @-mention is used as the message when sending notifications to the recipients.
- All text after the @-mention must identify the recipients. For example, in the screenshot above, the "Database" and "Hardware" xMatters groups are targeted for notification.
- You can use the same format to initiate the xmatters_conference webhook.
- If you want to specify an external conference bridge for an xmatters_conference webhook, add -bridge <bridge_name>, where <bridge_name> is a valid conference bridge already configured in xMatters. For example:
Please join the conference @webhook-xmatters_conference Database Hardware -bridge "My External Conference"

The Who's On Call feature displays current xMatters shift information within the Datadog event stream - no need to open a new browser window or login to xMatters.
For example, a DevOps team member wants to place a direct call to someone on the database team to confirm a solution. Instead of looking up the team in a company directory or calling a data center and hoping someone picks up, the DevOps person can use the Who's On Call webhook to see who is the right primary on-call person in the database group to contact.
When using the Who's On Call feature, you can use two different post types.
To test the Who's On Call feature:
- Click Events.
- In the Leave a status update field for an event, use the following format to initiate the webhook (see the Notes below for more syntax information):
@webhook-xmatters_on_call <group>
- Click Post.
- xMatters displays the Who's On Call results.
Notes
- Any text after the @-mention determines the type of Who's On Call you want to make.
- To get a group listing, leave the text after the @-mention empty or add -page x where x is a positive integer. The xMatters integration will reply with the page of groups for the value of x or, if empty, the first page of groups:
- To get on call shift details for a specific group, enter the name of a valid xMatters group after the @-mention. The xMattersintegration will reply with all active shifts for that group and the team members who are on call:
Troubleshooting
There are several places you can inspect when troubleshooting why an event is not sent to xMatters.

First, in Datadog, any errors in connecting to the URL encountered by the webhook create new events in the Datadog Event Stream. For example:
The failing webhook is highlighted in the payload, indicating that you are not authorized to access the @webhook-xmatters_engage feature.

The first place to look is the Activity Stream for the integration (click the gear icon and select Open Activity Stream). If there is an entry here, the request reached xMatters. Inspect the details for any errors.
For the two-way integration from xMatters to Datadog, the activity stream is the primary source of logging.
For example, if you are troubleshooting why a ticket was not assigned to a user after they replied with "Assign to me", check the Activity Steam for the notification responses:
Extending your integration
Looking to do more with xMatters and Datadog? If you want to tailor the settings and notifications for the integration, you can convert it to a custom workflow.

After you convert the integration to a custom workflow, you can configure it to use Basic authentication instead of URL authentication. If you switch the authentication from URL to Basic, you need to update the xMatters webhooks in Datadog to include an authorization header with the base64-encoded username and password of the integration user you created above.
You can create this value by following the instructions below:
- Go to https://www.base64encode.org/.
- In the “Encode to Base64 format” area, type the username and password of the integration user, separated by a colon (for example, datadog:5x>W!AkdfaHJd6092D)
- Click Encode, and copy the result.
Example: dXNlcm5hbWU6cGFzc3dvcmQ=
Update each xMatters webhook in Datadog by adding the following text to the Headers field, replacing <authentication_header_value> with the encoded result from above.
{
"Authorization": "Basic <authentication_header_value>"
}
Example: