Grafana is an open-source visualization and analytics software that lets you query, alert on, and understand your metrics by turning time-series data into graphs and visualizations. This workflow lets you send actionable alerts to on-call resources when xMatters gets a signal from Grafana. Responders can initiate an incident with the press of a button, or you can build on the flow to perform automated resolution tasks.
How it works
When an alert is generated in Grafana, it sends a JSON-formatted webhook to xMatters, based on user-defined alert rules. A Grafana trigger in xMatters parses the webhook and initiates a flow. The webook includes essential alert data that is included in notifications sent to users, and details about the origin of the alert that you can use to enrich the notifications or use when building automated tasks.
- Go to the Workflows Templates page and click the Grafana tile.
- On the Install Workflow dialog box, give the workflow a name (this must be unique in your instance) and add an optional description.
- You can edit these later, if needed.
- Click Install.
- After the workflow installs, the screen shows next steps as well the installation logs. The installation logs give you additional information about the installation, such as if you have languages in your instance that aren't configured for the messages in the workflow.
- Select the authentication method you want to use.
- Copy the trigger URL — you’ll use this to configure a webhook in Grafana.
- The trigger URL includes the recipients parameter, which specifies who should be notified. By default, this parameter is set to notify you (the logged in user), but you can set it to target any user or group you want.
- Click Done.
Configure Grafana to send requests to the trigger URL
To have Grafana send alerts to the flow trigger, you need to configure a webhook and set it to use the trigger URL.
- In Grafana, go to Alerting > Notification channels.
- Create a new Notification channel and fill in the following fields:
- Name: xMatters
- Type: Webhook
- Url: Paste the trigger URL you copied previously.
- Add the target names of any recipients you want xMatters to notify when the alert fires.
- For example, if you want to notify Emma Pearson, Mary McBride, and the on-call members in the Monitor Team responsible for the service, you'd add ?recipients=epearson,mmcbride,monitor%20team to the URL.
- You must URL-encode any special characters or spaces in the target names.
- Optional: If you choose to use Basic Authentication, expand the Optional Webhook settings menu to enter your Username and Password.
- Click Save.
- Add this notification channel to any alert rules you'd like to connect with xMatters.
You're ready to use the webhook to trigger automated flows, including steps such as sending alerts and initiating incidents, though we always recommend testing before putting things into use.
The trigger expects the recipients in the trigger URL. When you copy the URL from xMatters, it includes the recipients parameter: recipients=<yourname>. Of course, you don’t want to receive all the alerts.
To change the recipients for alerts from this webhook, swap out your name for the people or groups you want to target. For example, to target the 'DatabaseTeam', 'Antares Service Team', and 'HR & Marketing' groups, add recipients=databaseteam,antares%20service%20team,HR%20%26%20Marketing to the trigger URL. Remember to URL-encode any special characters, including spaces, in your group names.
When a condition you've set fires, it sends a signal to xMatters, which creates an event and notifies the individual or the on-call members of the people or groups you set as recipients in the webhook URL. When the trigger receives a signal saying the issue is resolved, it automatically terminates related events in xMatters.
The person responding to the notification has the following response options:
- Acknowledge: Acknowledges the notifications and stops escalations.
- Escalate: Immediately escalates the event to the next on-call resolver in a targeted group.
- Close: Ends the xMatters event and stops notifying all targeted recipients.
- Initiate Incident: Initiates an incident in xMatters.
Now that you've installed the workflow, you can use it as-is, or customize it to suit your needs better. Here are some examples of things you can add to the workflow to customize it:
- Use Slack, Zoom, and Microsoft Teams steps to add collaboration channels to the flow.
- Change the severity of incidents created when a recipient selects the Initiate Incident response.
- Update the message sent to resolvers to include the information most relevant to your team.
- Use the Grafana Alerts trigger to build your own custom flows.