Alert rules in Health Log Analytics
Summarize
Summary of Alert rules in Health Log Analytics
Health Log Analytics (HLA) automatically detects anomalies in log data by learning from patterns. However, for certain log types, especially low-frequency or critical logs, custom alert rules are necessary to reliably generate alerts. Custom alert rules allow you to specify metrics, thresholds, and alert properties tailored to your log data.
Show less
Key Features
- Log Pattern Classification: HLA classifies logs into three patterns to apply appropriate anomaly detection logic:
- Lively: Frequent logs (at least once every 20 seconds) use standard anomaly scoring with a reliable baseline.
- Sparse: Infrequent logs (less than once per minute) use probability distribution analysis, as standard scoring may be unreliable.
- Stopped: Logs that have stopped arriving for more than a configured period (default 5 minutes) can trigger alerts if the log stream was previously active.
- Custom Alert Rules: Enable precise alerting by defining conditions for metrics and thresholds that HLA may not detect automatically. This is especially important for:
- Low-frequency or periodic logs where automatic detection is unreliable.
- Known critical log messages that must always trigger alerts.
- Specific high-frequency log conditions where additional alerts are desired.
- Configurable System Properties: You can adjust detection timing and thresholds such as the dead signal period and alive period to fine-tune alerting behavior.
Practical Guidance for ServiceNow Customers
- For high-frequency logs with lively patterns, default anomaly detection is usually sufficient, but custom rules can add specificity.
- For low-frequency or periodic logs, create custom alert rules to ensure alerts are generated reliably.
- For critical conditions that must always trigger alerts, define custom rules to guarantee notification.
- Use custom alert rules to tailor alerting behavior to your unique log environment, improving operational awareness and response.
Next Steps
ServiceNow customers can manage custom alert rules through the following actions:
- Define new custom Log Analytics alert rules within Health Log Analytics.
- Modify existing custom alert rules to adapt to changing operational needs.
- Delete obsolete or unnecessary alert rules to maintain clarity and performance.
Health Log Analytics (HLA) detects anomalies automatically by learning from your log data. However, some log types require a custom alert rule to generate alerts reliably.
You can use custom alert rules to specify the metric, threshold, and alert properties for generating alerts that HLA might not detect automatically.
Anomaly detection logic by log pattern
HLA classifies incoming logs into three patterns before applying anomaly detection. This classification determines which detection logic is used.
| Pattern | Description |
|---|---|
| Lively | Logs arrive frequently and consistently: at least once in 20 seconds. For example, an application writes hundreds of log entries per hour. The ML Engine has enough volume to build a reliable baseline, so it applies standard anomaly scoring to detect deviations. |
| Sparse | Logs arrive infrequently or irregularly: less than once in 1 minute (60 seconds). For example, a batch job runs every night and writes a small number of log entries. Standard anomaly scoring would produce unreliable results here, so the ML Engine applies probability distribution analysis instead. Sparse logs might not generate alerts if the volume is too low to establish a baseline. |
| Stopped | Logs have not arrived for more than the configured period. Default is 5 minutes (300 seconds). To modify the default value, update the HLA system property detective.resolution.signal_dead.For a log stream to be alerted as stopped or dead it must first be considered alive by running continuously for
a minimum period of time. You can set this time in the HLA system property For information about setting and changing HLA system properties, see Configure global Health Log Analytics system properties. |
When to create custom alert rules
- For high-frequency logs with a lively log pattern, there is no need for a custom rule. However, you can add a rule to generate alerts under specific conditions.
- For low-frequency logs with a sparse log pattern, the system might not generate alerts automatically. If these logs should still generate alerts, define a custom alert rule.
- For known critical conditions that HLA might not flag automatically, define a custom rule. For example, if a specific log message indicates that a critical service has failed, define a rule that generates an alert every time that message appears.
| Scenario | ML detection | Custom rule |
|---|---|---|
| High-frequency logs with a lively log pattern | Likely sufficient | Optional |
| Low-frequency or periodic logs | Unreliable | Suggested |
| Known critical conditions | Insufficient | Required |