Alert rules

Alert rules are how you stop watching the monitor. A rule watches a metric or condition and fires when it crosses a threshold you set.

Creating a rule

A rule needs four things:

1. Scope — which channel or channels it applies to.
2. Condition — the metric and threshold. Error rate above a percentage, queue depth above a count, a channel entering an errored state.
3. Severity — how loudly this should be treated.
4. Notification — where it goes, using a notification template.

Severities

Severity is the field people skip, and then everything is urgent and nothing is. Reserve the top severity for conditions that need someone awake — a production clinical feed stopped — and use lower severities for things that can wait for the morning.

Acknowledge and mute

Acknowledge an active alert to signal that a human is on it. The acknowledgement, and who made it, is recorded.

Mute a rule when you already know. During a planned downstream outage, muting the relevant rule for the window keeps the noise down without deleting the rule and forgetting to recreate it.

Alert history shows what fired, when, who acknowledged it, and how long it stayed open.

Notification templates

Templates control what the notification actually says. It is worth putting the channel name, the environment and the condition in the first line — an alert that reads well at 3am is doing most of its job.

Plan limits

Plan	Alert rules
Free	5
Team	25
Enterprise	500

Rule counts are per organization. If you are up against the limit on Team, it is usually a sign that rules are scoped per channel where a rule scoped to a set of channels would do.

What to alert on first

If you are starting from nothing, three rules cover most of the value:

• Production channel entered an errored state.
• Error rate above a threshold sustained over several minutes.
• Queue depth above a count that would take more than a few minutes to clear.