Event Flood Control
During an event flood, the first events to arrive typically contain the details you need to identify which of your systems need attention. Subsequent events that are generated during a flood add a lot of extra noise and they can result in processing delays if you exceed your licensed number of events per minute.
Event flood control reduces the number of events that are generated in xMatters when you experience a flood or event storm from one or more of your systems.
How does it work?
This feature compares incoming event requests to recent events and suppresses correlated events that occur in too close succession to one another. When suppression kicks in, xMatters sends you a notification and logs the details of the flood on the Events report so you can track how many events were suppressed and why. These suppressed events aren't queued for processing and don't count against your licensed number of events per minute, which allows events from other sources to flow through xMatters without lengthy processing delays.
A default Event Rate Filter rule applies to all new and existing inbound integrations. The default rule automatically suppresses incoming events from the same integration that target the same recipients when they occur at a rate that exceeds four events per minute.
This default rule is based on our analysis of actual customer event flood data and provides a balance of effective protection against sudden influxes of events, while allowing real traffic to proceed as normal.
The Event Rate Filter rule is designed to allow a reasonable number of events to be created in xMatters to indicate the type of activity that's occurring. This prompts users to view the source system for more information. In all customer cases that we reviewed, a recipient receiving more than four events per minute about an incident is able to gain better situational awareness by going to the source system and evaluating it there.
For more information on how you can edit the default rules, or create your own event flood control rules, see Manage Event Flood Control Settings.
If events from an integration with the same recipients exceed the rate specified in the Event Rate Filter, event flood control is triggered and the additional event requests are suppressed beneath the most recently processed event to which they correlate (the "parent" event). This is indicated on the Events report by a stacked icon with a running badge count of the number of suppressed events.
To view more information about suppressed events, click the stacked icon below the parent event to view the Suppression report.
You can also use the xMatters REST API to retrieve data about events with a "SUPPRESSED" status using the GET /events endpoint.
Suppression remains in effect until the incoming rate of correlated event requests drops below the rate specified by the trigger condition (e.g., four or fewer events per minute for the default rule). When the flood is over, xMatters resumes generating events as normal and lists them in the Recent Events report.
When event flood control is triggered, the recipients who would otherwise have received a flood of event notifications receive a single notification from xMatters to inform them that an event flood was detected. They also receive periodic updates while the flood is ongoing.
These 'Event flood detected' notifications are associated with system events initiated by xMatters, which you'll see listed in the Recent Events report.
Notifications that xMatters sends to inform recipients that a flood has occurred or is ongoing include the following information:
- The name of the workflow and form used to create the event.
- The name of the event flood control rule that triggered event suppression.
- The time the first event was suppressed.
- The total number of events that have been suppressed so far.
- The event ID of the parent event.
The notification includes the following response options:
- Acknowledge: Stops notifying other users about the flood.
- Close: Stops notifying all recipients and terminates the flood notification event.
- Escalate: Stops notifying the current user and immediately escalates the event to the next scheduled recipient.
As long as the flood continues to meet the conditions of the event flood control rule, xMatters will initiate a new system event and send an updated version of the notification according to specified notification interval (by default, every 15 minutes, or for every 1000 suppressed events, whichever occurs first).
During a flood, xMatters automatically notifies and updates the targeted recipients of the event that triggered the flood. If you're not a targeted recipient and want to be informed when event floods occur and are ongoing in your system, you can subscribe to receive these notifications.
xMatters includes an internal workflow with a built-in one-way FYI subscription for event floods. When you create a subscription, you can select a specific form or integration that you'd like to receive event flood notifications about.
To subscribe to event flood control alerts:
- Click your user name at the top-right corner of the page. A drop-down menu appears.
- In the drop-down menu, click Subscriptions.
- On the Subscriptions page, click Add Subscription, and from the drop-down list select Event Flood Detected.
- On the Subscription Details page, give your subscription a name and description.
- Set the planName and integrationName criteria to specify the name of the workflow and integration to which you want to subscribe.
- To be notified about event floods from any of the integrations in a workflow, specify the planName and leave the integrationName field blank.
- Set the remaining options to customize the subscription, and then click Subscribe.
xMatters will now send you a notification whenever event flood control is triggered for the specified integration, and updates if the flood is ongoing.
The Event Flood Control page in the web user interface allows you to view and manage event flood control rules for each event source.
To manage event flood control rules for a workflow or built in integration, you require the following:
- A role with permission to manage event flood control settings (this includes Company Supervisors, Full Access Users, and REST Web Service Users).
- Access permissions to the relevant workflow or built-in integration.
To view your event flood control rules:
- Click the Workflows tab.
- In the menu, click Flood Control.
The following table describes the information displayed on the Event Flood Control page.
The name of the event flood control rule. xMatters automatically creates a default "Event Rate Filter" for each of your inbound integrations.
The properties used to correlate event requests and the rate of incoming event requests at which suppression begins. For the default filter, this includes events targeting the same integration and same recipients more than four times in one minute.
Click on the field's value to view the full list of properties and when suppression starts:
If you have permission to edit the settings for a rule, an Edit Rule hyperlink is visible. Click this link to navigate to the rule's configuration screen.
The workflow or built in integration whose inbound integrations the rule applies to.
|Last Occurrence||The date and time the rule last suppressed events.|
|Status||A check box to enable or disable the rule.|
To more easily view data in the table, do one or more of the following:
- To sort the content in alphabetical or chronological order based on a specific column, click the column header. (Click the header again to reverse the sort order.)
- To change the width of a column, move the pointer over the edge of the column and when the re-size icon appears, click and drag the column to the desired size.
- To rearrange the columns, click and drag a column header to a new location on the table.
- If the list contains more entries or columns than can be displayed on your screen, scroll horizontally or vertically. The system automatically loads more data as scroll down the page.
To enable or disable a flood control rule, do one of the following:
- From the main Event Flood Control page, select or clear the check box in the Status column that corresponds to that rule.
- From the configuration page for a rule, select or clear the check box at the top of the screen. This check box appears after you save your event flood control rule.
If the default Event Rate Filter doesn't suit your needs, you have the option to create your own event flood control rules. This allows you to customize the event properties and trigger conditions used for event suppression, as well as the settings for event flood control notifications.
For example, if your system aggregates event traffic before sending it to xMatters, you can create rules with different properties to apply event flood control separately for your individual source systems.
To create a new event flood control rule:
- In xMatters, access the Flood Control page.
- At the top of the Flood Control page, click Create Rule.
- In the pop-up window that appears, give your rule a unique name and select the source of events it applies to, and then click Create Rule.
- The name of a rule must be unique within a workflow.
- Select the event source for your rule.
- Source: This field is pre-populated with the workflow or built-in integration you selected when you created your rule. You can edit the source before you save your rule.
- Integration: The value for this field defaults to "All Integrations", or you can select a specific integration in your workflow or built-in integration.
Built-in integrations have a streamlined configuration process that is supported by an underlying workflows, which may consist of multiple forms and integrations. If you select a built-in integration as your source, the Integration drop-down will contain the names of the individual integrations the configuration is based on.
- Select the event properties that you want xMatters to use to determine if incoming event requests are correlated. If incoming event requests have matching values for the selected properties, the rule will be triggered.
- Drag properties from the list of available properties to the Selected Properties column to include them in your rule.
- The list of available properties includes any properties that you've created in your workflow, and a set of system properties that are available for every workflow (Recipients, Priority, Initiator, Incident, and Integration).
- To remove a property from the selected properties list, click the "X" next to its name. This will return the property to the list of available properties.
- Set the trigger conditions for your rule. When these conditions are met, event suppression will begin.
- This setting inherits the values of the default Event Rate Filter, which triggers suppression when the rate of incoming event requests is greater than 4 events within 60 seconds.
- You may specify between 1 to 999 events and 10 to 3600 seconds for the trigger conditions.
- Set the notification interval to notify recipients if the event flood is still active after a set amount of time or number of suppressed events, whichever comes first.
- This setting inherits the values of the default Event Rate Filter, which notifies recipients of an ongoing flood every 900 seconds (15 minutes), or 1000 events.
- You may select to notify users between 60 to 43200 seconds (1 minute to 12 hours), or for every 5 to 10000 events.
- Optionally, you can send a webhook to a specific URL over the duration of an event flood. This can be useful if, for example, you want to send a list of suppressed events back to the source of the signals so that system can mark the events as closed. The webhook payload is described in the xMatters REST API under Event Suppressions. Specify an interval in minutes, and provide the URL where you want to send the webhook.
- Click Save.
Once you save your event flood control rule, you cannot edit the Source or Integration fields that define your event source.
To access the configuration screen for a flood control rule and edit its settings, do one of the following:
- Click the name of the rule on the event flood control page. If you do not have permission to edit a rule, its name will not be a hyperlink.
- Click Edit Rule from the rule's property list on the Event Flood Control page, or on the Suppression report. If you do not have permission to edit the rule, this text will not appear beneath the list of properties.
For more information about the configuration settings for an event flood control rule, see Create a rule.
Select the check box next to the name of the rule, and then click Delete. The Delete button only appears after you select a rule from the list.