Event Flood Control
During an event flood, the first events to arrive typically contain the details you need to identify which of your systems need attention. Subsequent events that are generated during a flood add a lot of extra noise and they can result in processing delays if you exceed your licensed number of events per minute.
Event flood control reduces the number of events that are generated in xMatters when you experience a flood or event storm from one or more of your systems.
How does it work?
This feature compares incoming event requests to recent events and suppresses correlated events that occur in too close succession to one another. When suppression kicks in, xMatters sends you a notification and logs the details of the flood on the Events report so you can track how many events were suppressed and why. These suppressed events aren't queued for processing and don't count against your licensed number of events per minute, which allows events from other sources to flow through xMatters without lengthy processing delays.
Default Event Rate Filter
As of the Defender release of xMatters On Demand, a default Event Rate Filter applies to all new and existing inbound integrations. The default event rate filter automatically suppresses incoming events from the same integration that target the same recipients when they occur at a rate that exceeds four events per minute.
This default rule is based on our analysis of actual customer event flood data and provides a balance of effective protection against sudden influxes of events, while allowing real traffic to proceed as normal. For more information on how you can enable or disable the default rule for an integration, see Manage Event Flood Control Settings.
If events from an integration with the same recipients exceed the rate specified in the Event Rate Filter, event flood control is triggered and the additional event requests are suppressed beneath the most recently processed event to which they correlate (the "parent" event). This is indicated on the Events report by a stacked icon with a running badge count of the number of suppressed events.
To view more information about suppressed events, click the stacked icon below the parent event to view the Suppression report.
You can also use the xMatters REST API to retrieve data about events with a "SUPPRESSED" status using the GET /events endpoint.
Suppression remains in effect until the incoming rate of correlated event requests drops to four or fewer events per minute. When the flood is over, xMatters resumes generating events as normal and lists them in the Recent Events report.
When event flood control is triggered, the recipients who would otherwise have received a flood of event notifications receive a single notification from xMatters to inform them that an event flood was detected. They also receive periodic updates while the flood is ongoing.
These 'Event flood detected' notifications are associated with system events initiated by xMatters, which you'll see listed in the Recent Events report.
Notifications that xMatters sends to inform recipients that a flood has occurred or is ongoing include the following information:
- The name of the communication plan and form used to create the event.
- The name of the event flood control filter that triggered event suppression.
- The time the first event was suppressed.
- The total number of events that have been suppressed so far.
- The event ID of the parent event.
The notification includes the following response options:
- Acknowledge: Stops notifying other users about the flood.
- Close: Stops notifying all recipients and terminates the flood notification event.
- Escalate: Stops notifying the current user and immediately escalates the event to the next scheduled recipient.
As long as the flood continues to meet the conditions of the Event Rate filter, xMatters will initiate a new system event and send an updated version of the notification every 15 minutes, or for every 1000 suppressed events, whichever occurs first.
During a flood, xMatters automatically notifies and updates the targeted recipients of the event that triggered the flood. If you're not a targeted recipient and want to be informed when event floods occur and are ongoing in your system, you can subscribe to receive these notifications.
xMatters includes an internal communication plan with a built-in one-way FYI subscription for event floods. When you create a subscription, you can select a specific communication plan or integration that you'd like to receive event flood notifications about.
To subscribe to event flood control alerts:
- Click your user name at the top-right corner of the page. A drop-down menu appears.
- In the drop-down menu, click Subscriptions.
- On the Subscriptions page, click Add Subscription, and from the drop-down list select Event Flood Detected.
- On the Subscription Details page, give your subscription a name and description.
- Set the planName and integrationName criteria to specify the name of the communication plan and integration to which you want to subscribe.
- To be notified about event floods from any of the integrations in a communication plan, specify the planName and leave the integrationName field blank.
- Set the remaining options to customize the subscription, and then click Subscribe.
xMatters will now send you a notification whenever event flood control is triggered for the specified integration, and updates if the flood is ongoing.
Manage Event Flood Control Settings
The Defender release of xMatters On-Demand introduces a new Event Flood Control page in the web user interface, which allows you to view and manage event flood control rules for individual communication plans and built-in integrations.
To view your company's event flood control rules:
- Click the Developer tab.
- In the Communication Plan Builder menu, click Flood Control.
- By default, the flood control interface displays the Event Flood Control tab.
- On the other tab, you can access your company's Notification Flood Control settings.
The following table describes the information displayed on the Event Flood Control page.
|Name||The name of the event flood control rule. The initial release of this feature creates the default "Event Rate Filter" for each of your new or existing communication plans and built-in integrations.|
The properties used to correlate event requests and the rate of incoming event requests at which suppression begins. For the default filter, this includes events targeting the same integration and same recipients more than four times in one minute.
Click on the field's value to view the full list of properties and when suppression starts:
The communication plan or built in integration whose inbound integrations the rule applies to.
|Last Occurrence||The date and time the rule last suppressed events.|
|Status||A check box to enable or disable the rule.|
To more easily view data in the table, do one or more of the following:
- To sort the content of a column in alphabetical or chronological order, click the column header. (Click the header again to reverse the sort order.)
- To change the width of a column, move the pointer over the edge of the column and when the re-size icon appears, click and drag the column to the desired size.
- To rearrange the columns, click and drag a column header to a new location on the table.
- If the list contains more entries or columns than can be displayed on your screen, scroll horizontally or vertically. The system automatically loads more data as scroll down the page.
To enable, disable, or delete event flood control rules for a communication plan or built in integration, you require the following permissions:
- A role which has permission to manage event flood control settings (this includes Company Supervisors and Full Access Users).
- Access permissions to the relevant communication plan or built-in integration.
To enable or disable an event flood control rule:
Select or clear the check box in the Status column that corresponds to that rule.
To delete an event flood control rule:
Select the check box next to the name of the rule, and then click Delete. The Delete button only appears after you select a rule from the list.
In the initial release of this feature, deleting the Event Rate Filter for a communication plan or built-in integration is a permanent action; once the filter is deleted, events from integrations in that communication plan or built-in integration are no longer suppressed during an event flood. You'll be able to create event flood control filters for existing integrations an upcoming release of xMatters On-Demand.