Example observability metrics alerts

The following examples show how observability metrics alerts behave under different trigger conditions. Each example walks through configuring an alert definition, the observed metric across an evaluation period, and the point at which the alert fires and resolves. For background on how alerts are triggered and resolved, check out our How alerting works guide.

Threshold-based alerts

Threshold-based alerts trigger when the observed metric goes above or below a static numeric threshold. You can configure them to fire on any single minute that exceeds the threshold, on consecutive minutes that fall below the threshold, or on all minutes of an evaluation period that exceed the threshold.

Above a static threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 5XX responses exceed 1,000 on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

  • the Data source menu to Historic Stats
  • the Metric menu to status_5xx
  • the Evaluation type menu to Above (one minute)
  • the Threshold field to 1000
  • the Evaluation period menu to 5 minutes
  • the Integrations menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes above 1,000 (the Threshold):

Evaluation period minuteNumber of 5XX responsesThreshold exceeded
1988No
2765No
31021Yes
4843No
51000No

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the third minute of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell Alert bell icon that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there are zero one minute intervals in the evaluation period that exceed the threshold:

Evaluation period minuteNumber of 5XX responsesThreshold exceeded
1576No
2674No
3516No
4577No
5732No

Below a static threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 2XX responses is below 500 on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

  • the Data source menu to Historic Stats
  • the Metric menu to status_2xx
  • the Evaluation type menu to Below (two minutes)
  • the Threshold field to 500
  • the Evaluation period menu to 5 minutes
  • the Integrations menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During two consecutive polling minutes in one of those evaluation periods, the observed metric is below 500 (the Threshold):

Evaluation period minuteNumber of 2XX responsesBelow the threshold
11098No
21254No
3372Yes
4415Yes
5500No

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric was below the threshold during the third and fourth minutes of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell Alert bell icon that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there aren't two consecutive minutes in the evaluation period that are below the threshold:

Evaluation period minuteNumber of 2XX responsesBelow the threshold
11108No
2285Yes
31284No
41065No
5396Yes

Above a static threshold for all minutes alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 5XX responses exceed 1,000 on a single service or all your services over the entire evaluation period. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

  • the Data source menu to Historic Stats
  • the Metric menu to 5XX
  • the Alert type controls to Threshold
  • the Evaluation type menu to Above (all minutes)
  • the Threshold field to 1000
  • the Evaluation period menu to 5 minutes
  • the Integrations menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During an evaluation period and polling every minute, the observed metric goes above 1,000 (the Threshold) at every minute:

Evaluation period minuteNumber of 5XX responsesThreshold exceeded
11001Yes
21110Yes
31021Yes
41050Yes
51032Yes

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the entire evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell Alert bell icon that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.

That alert then resolves when at least one minute in the evaluation period no longer exceeds the threshold:

Evaluation period minuteNumber of 5XX responsesThreshold exceeded
1900No
21001Yes
3516No
4577No
5732No

Rate-based alerts

Rate-based alerts trigger when the observed metric goes above or below a percentage of overall requests. You can configure them to fire on any single minute that exceeds the rate or on consecutive minutes that fall below the rate.

Above a rate-based threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 4XX responses exceeds 5% of all responses sent by a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

  • the Data source menu to Historic Stats
  • the Metric menu to 4XX Error Rate
  • the Evaluation type menu to Above (one minute)
  • the Threshold menu to 5%
  • the Evaluation period menu to 5 minutes
  • the Integrations menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes above 5% (the Threshold):

Evaluation period minuteNumber of 4XX responsesTotal number of requestsPercentage of overall requestsThreshold exceeded
109880%No
2137651.67%No
36410216.27%Yes
4148431.66%No
5310000.3%No

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the third minute of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell Alert bell icon that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there are zero one minute intervals in the evaluation period that exceed the threshold:

Evaluation period minuteNumber of 4XX responsesTotal number of requestsPercentage of overall requestsThreshold exceeded
1195763.30%No
2306744.45%No
3145162.71%No
405770%No
507320%No

Below a rate-based threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the Success Rate, that is, the percentage of responses with non-5XX codes, falls below 80% on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

  • the Data source menu to Historic Stats
  • the Metric menu to Success Rate
  • the Evaluation type menu to Below (two minutes)
  • the Threshold field to 80%
  • the Evaluation period menu to 5 minutes
  • the Integrations menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes below 80% (the Threshold):

Evaluation period minuteSuccess RateBelow the threshold
199%No
279%Yes
375%Yes
495%No
590%No

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric was below the threshold during two consecutive minutes in the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell Alert bell icon that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there aren't two consecutive minutes in the evaluation period that are below the threshold:

Evaluation period minuteSuccess RateBelow the threshold
190%No
295%No
399%No
499%No
589%No

Percentage change alerts

Percentage change alerts trigger when the observed metric increases by, decreases by, or changes in either direction by a specified percentage. The change is calculated by comparing the first and last minute of the comparison period.

Percentage increase alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 4XX responses increases by 5% on a single service excluding 4XX values less than 10. When configuring the alert definition, you choose to report on a single service. Then, you set:

  • the Data source menu to Historic Stats
  • the Metric menu to 4XX
  • the Alert type controls to Percentage change
  • the Evaluation type menu to Increases by
  • the Percentage field to 5%
  • the Comparison period menu to 5 minutes
  • the Excluded values field to 10
  • the Integrations menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Comparison period). During a comparison period, the observed metric percentage calculation increases by 5%:

Comparison period minuteNumber of 4XX responsesComparison usedExcluded?
19NoYes
215YesNo
314NoNo
413NoNo
512NoNo
625YesNo

At the sixth minute and the end of the five minute comparison period, an alert is triggered because the observed metric and percentage calculation increased by at least 5% (25 - 15 / 25 = 0.4 x 100 = 40% > 5%). Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell Alert bell icon that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.

That alert then resolves when the percentage calculation conditions are no longer met.

Percentage decrease alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 2XX responses decreases by 40% on a single service. When configuring the alert definition, you choose to report on a single service. Then, you set:

  • the Data source menu to Historic Stats
  • the Metric menu to 2XX
  • the Alert type controls to Percentage change
  • the Evaluation type menu to Decreases by
  • the Percentage field to 40%
  • the Comparison period menu to 5 minutes
  • the Integrations menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Comparison period). During a comparison period, the observed metric percentage calculation decreases by 40%:

Comparison period minuteNumber of 2XX responsesComparison used
1100No
290No
385No
470No
560Yes

At the end of the fifth minute of that comparison period, an alert is triggered because the observed metric and percentage calculation decreased by at least 40% (60 - 100 / 100 = 0.4 x 100 = 40%). Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell Alert bell icon that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.

That alert then resolves when the percentage calculation conditions are no longer met.

Percentage increase or decrease alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 4XX responses changes by 20% in either direction on a single service. When configuring the alert definition, you choose to report on a single service. Then, you set:

  • the Data source menu to Historic Stats
  • the Metric menu to 4XX
  • the Alert type controls to Percentage change
  • the Evaluation type menu to Increases or decreases by
  • the Percentage field to 20%
  • the Comparison period menu to 5 minutes
  • the Integrations menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Comparison period). During a comparison period, the observed metric percentage calculation changes by at least 20%:

Comparison period minuteNumber of 4XX responsesComparison used
1100Yes
295No
3110No
4125No
5135Yes

At the end of the fifth minute of that comparison period, an alert is triggered because the observed metric and percentage calculation increased by at least 20% (135 - 100 / 100 = 0.35 x 100 = 35%). Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell Alert bell icon that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.

That alert then resolves when the percentage calculation conditions are no longer met.