Example observability metrics alerts
The following examples show how observability metrics alerts behave under different trigger conditions. Each example walks through configuring an alert definition, the observed metric across an evaluation period, and the point at which the alert fires and resolves. For background on how alerts are triggered and resolved, check out our How alerting works guide.
Threshold-based alerts
Threshold-based alerts trigger when the observed metric goes above or below a static numeric threshold. You can configure them to fire on any single minute that exceeds the threshold, on consecutive minutes that fall below the threshold, or on all minutes of an evaluation period that exceed the threshold.
Above a static threshold alert example
Let's say you want to be notified in your #website-alerts Slack channel when the number of 5XX responses exceed 1,000 on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:
- the Data source menu to
Historic Stats - the Metric menu to
status_5xx - the Evaluation type menu to
Above (one minute) - the Threshold field to
1000 - the Evaluation period menu to
5 minutes - the Integrations menu to
website-alerts
After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes above 1,000 (the Threshold):
| Evaluation period minute | Number of 5XX responses | Threshold exceeded |
|---|---|---|
| 1 | 988 | No |
| 2 | 765 | No |
| 3 | 1021 | Yes |
| 4 | 843 | No |
| 5 | 1000 | No |
At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the third minute of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell 
that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.
That alert then resolves at the end of this evaluation period because there are zero one minute intervals in the evaluation period that exceed the threshold:
| Evaluation period minute | Number of 5XX responses | Threshold exceeded |
|---|---|---|
| 1 | 576 | No |
| 2 | 674 | No |
| 3 | 516 | No |
| 4 | 577 | No |
| 5 | 732 | No |
Below a static threshold alert example
Let's say you want to be notified in your #website-alerts Slack channel when the number of 2XX responses is below 500 on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:
- the Data source menu to
Historic Stats - the Metric menu to
status_2xx - the Evaluation type menu to
Below (two minutes) - the Threshold field to
500 - the Evaluation period menu to
5 minutes - the Integrations menu to
website-alerts
After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During two consecutive polling minutes in one of those evaluation periods, the observed metric is below 500 (the Threshold):
| Evaluation period minute | Number of 2XX responses | Below the threshold |
|---|---|---|
| 1 | 1098 | No |
| 2 | 1254 | No |
| 3 | 372 | Yes |
| 4 | 415 | Yes |
| 5 | 500 | No |
At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric was below the threshold during the third and fourth minutes of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.
That alert then resolves at the end of this evaluation period because there aren't two consecutive minutes in the evaluation period that are below the threshold:
| Evaluation period minute | Number of 2XX responses | Below the threshold |
|---|---|---|
| 1 | 1108 | No |
| 2 | 285 | Yes |
| 3 | 1284 | No |
| 4 | 1065 | No |
| 5 | 396 | Yes |
Above a static threshold for all minutes alert example
Let's say you want to be notified in your #website-alerts Slack channel when the number of 5XX responses exceed 1,000 on a single service or all your services over the entire evaluation period. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:
- the Data source menu to
Historic Stats - the Metric menu to
5XX - the Alert type controls to
Threshold - the Evaluation type menu to
Above (all minutes) - the Threshold field to
1000 - the Evaluation period menu to
5 minutes - the Integrations menu to
website-alerts
After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During an evaluation period and polling every minute, the observed metric goes above 1,000 (the Threshold) at every minute:
| Evaluation period minute | Number of 5XX responses | Threshold exceeded |
|---|---|---|
| 1 | 1001 | Yes |
| 2 | 1110 | Yes |
| 3 | 1021 | Yes |
| 4 | 1050 | Yes |
| 5 | 1032 | Yes |
At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the entire evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.
That alert then resolves when at least one minute in the evaluation period no longer exceeds the threshold:
| Evaluation period minute | Number of 5XX responses | Threshold exceeded |
|---|---|---|
| 1 | 900 | No |
| 2 | 1001 | Yes |
| 3 | 516 | No |
| 4 | 577 | No |
| 5 | 732 | No |
Rate-based alerts
Rate-based alerts trigger when the observed metric goes above or below a percentage of overall requests. You can configure them to fire on any single minute that exceeds the rate or on consecutive minutes that fall below the rate.
Above a rate-based threshold alert example
Let's say you want to be notified in your #website-alerts Slack channel when the number of 4XX responses exceeds 5% of all responses sent by a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:
- the Data source menu to
Historic Stats - the Metric menu to
4XX Error Rate - the Evaluation type menu to
Above (one minute) - the Threshold menu to
5% - the Evaluation period menu to
5 minutes - the Integrations menu to
website-alerts
After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes above 5% (the Threshold):
| Evaluation period minute | Number of 4XX responses | Total number of requests | Percentage of overall requests | Threshold exceeded |
|---|---|---|---|---|
| 1 | 0 | 988 | 0% | No |
| 2 | 13 | 765 | 1.67% | No |
| 3 | 64 | 1021 | 6.27% | Yes |
| 4 | 14 | 843 | 1.66% | No |
| 5 | 3 | 1000 | 0.3% | No |
At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the third minute of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.
That alert then resolves at the end of this evaluation period because there are zero one minute intervals in the evaluation period that exceed the threshold:
| Evaluation period minute | Number of 4XX responses | Total number of requests | Percentage of overall requests | Threshold exceeded |
|---|---|---|---|---|
| 1 | 19 | 576 | 3.30% | No |
| 2 | 30 | 674 | 4.45% | No |
| 3 | 14 | 516 | 2.71% | No |
| 4 | 0 | 577 | 0% | No |
| 5 | 0 | 732 | 0% | No |
Below a rate-based threshold alert example
Let's say you want to be notified in your #website-alerts Slack channel when the Success Rate, that is, the percentage of responses with non-5XX codes, falls below 80% on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:
- the Data source menu to
Historic Stats - the Metric menu to
Success Rate - the Evaluation type menu to
Below (two minutes) - the Threshold field to
80% - the Evaluation period menu to
5 minutes - the Integrations menu to
website-alerts
After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes below 80% (the Threshold):
| Evaluation period minute | Success Rate | Below the threshold |
|---|---|---|
| 1 | 99% | No |
| 2 | 79% | Yes |
| 3 | 75% | Yes |
| 4 | 95% | No |
| 5 | 90% | No |
At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric was below the threshold during two consecutive minutes in the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.
That alert then resolves at the end of this evaluation period because there aren't two consecutive minutes in the evaluation period that are below the threshold:
| Evaluation period minute | Success Rate | Below the threshold |
|---|---|---|
| 1 | 90% | No |
| 2 | 95% | No |
| 3 | 99% | No |
| 4 | 99% | No |
| 5 | 89% | No |
Percentage change alerts
Percentage change alerts trigger when the observed metric increases by, decreases by, or changes in either direction by a specified percentage. The change is calculated by comparing the first and last minute of the comparison period.
Percentage increase alert example
Let's say you want to be notified in your #website-alerts Slack channel when the number of 4XX responses increases by 5% on a single service excluding 4XX values less than 10. When configuring the alert definition, you choose to report on a single service. Then, you set:
- the Data source menu to
Historic Stats - the Metric menu to
4XX - the Alert type controls to
Percentage change - the Evaluation type menu to
Increases by - the Percentage field to
5% - the Comparison period menu to
5 minutes - the Excluded values field to
10 - the Integrations menu to
website-alerts
After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Comparison period). During a comparison period, the observed metric percentage calculation increases by 5%:
| Comparison period minute | Number of 4XX responses | Comparison used | Excluded? |
|---|---|---|---|
| 1 | 9 | No | Yes |
| 2 | 15 | Yes | No |
| 3 | 14 | No | No |
| 4 | 13 | No | No |
| 5 | 12 | No | No |
| 6 | 25 | Yes | No |
At the sixth minute and the end of the five minute comparison period, an alert is triggered because the observed metric and percentage calculation increased by at least 5% (25 - 15 / 25 = 0.4 x 100 = 40% > 5%). Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.
That alert then resolves when the percentage calculation conditions are no longer met.
Percentage decrease alert example
Let's say you want to be notified in your #website-alerts Slack channel when the number of 2XX responses decreases by 40% on a single service. When configuring the alert definition, you choose to report on a single service. Then, you set:
- the Data source menu to
Historic Stats - the Metric menu to
2XX - the Alert type controls to
Percentage change - the Evaluation type menu to
Decreases by - the Percentage field to
40% - the Comparison period menu to
5 minutes - the Integrations menu to
website-alerts
After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Comparison period). During a comparison period, the observed metric percentage calculation decreases by 40%:
| Comparison period minute | Number of 2XX responses | Comparison used |
|---|---|---|
| 1 | 100 | No |
| 2 | 90 | No |
| 3 | 85 | No |
| 4 | 70 | No |
| 5 | 60 | Yes |
At the end of the fifth minute of that comparison period, an alert is triggered because the observed metric and percentage calculation decreased by at least 40% (60 - 100 / 100 = 0.4 x 100 = 40%). Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.
That alert then resolves when the percentage calculation conditions are no longer met.
Percentage increase or decrease alert example
Let's say you want to be notified in your #website-alerts Slack channel when the number of 4XX responses changes by 20% in either direction on a single service. When configuring the alert definition, you choose to report on a single service. Then, you set:
- the Data source menu to
Historic Stats - the Metric menu to
4XX - the Alert type controls to
Percentage change - the Evaluation type menu to
Increases or decreases by - the Percentage field to
20% - the Comparison period menu to
5 minutes - the Integrations menu to
website-alerts
After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Comparison period). During a comparison period, the observed metric percentage calculation changes by at least 20%:
| Comparison period minute | Number of 4XX responses | Comparison used |
|---|---|---|
| 1 | 100 | Yes |
| 2 | 95 | No |
| 3 | 110 | No |
| 4 | 125 | No |
| 5 | 135 | Yes |
At the end of the fifth minute of that comparison period, an alert is triggered because the observed metric and percentage calculation increased by at least 20% (135 - 100 / 100 = 0.35 x 100 = 35%). Notifications are also sent. Specifically, the Fastly application indicates via the Alert bell that an observability alert has been triggered and a message is sent to the #website-alerts Slack channel.
That alert then resolves when the percentage calculation conditions are no longer met.
