Monitoring log data flow and optimizing integration settings in Health Log Analytics

  • Release version: Yokohama
  • Updated July 31, 2025
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Monitoring Log Data Flow and Optimizing Integration Settings in Health Log Analytics

    The Overview screen in Health Log Analytics provides a detailed view of the log-processing pipeline for active integrations, allowing you to monitor log data streaming status, troubleshoot issues, and adjust integration settings as necessary. It connects you to various components, including Data Input Mapping and Log Viewer, providing contextual insight into the current integration.

    Show full answer Show less

    Key Features

    • Streaming Status: Displays real-time data on streaming status between the log source, MID Server, and HLA engine, including logs per minute metrics. For certain integrations, it also indicates failures and suggests corrective actions.
    • Log Streaming Sources: Shows detailed information about integration log sources, including status, last event time, and average log line rates, enabling you to assess data flow effectively.
    • Auto-refresh & Alerts: Live data updates every minute, with customizable refresh intervals and warning messages for non-streaming integrations.

    Key Outcomes

    By effectively using the Overview screen, customers can ensure continuous data flow into ServiceNow, vital for anomaly detection and alert generation. In case of streaming failures, customers can quickly identify and resolve issues, such as credential errors, restoring service functionality promptly. This proactive monitoring capability enhances operational efficiency and reliability in managing log data integrations.

    The Overview screen in Health Log Analytics provides a comprehensive view of the components in the log-processing pipeline of active integrations. From this screen, you can troubleshoot any streaming issues and adjust the integration’s settings if needed.

    The Overview screen shows the log data streaming status and streaming sources of an active integration. It provides direct access to the Data Input Mapping, Source Type Structures, and Log Sources pages, as well as the Log Viewer, all with context from the current integration.

    Figure 1. Overview screen and View menu items
    Integration Overview screen and View menu items.

    Streaming status

    The Streaming status section on the Overview screen shows the status of data streaming between the log source and the MID Server and between the MID Server and the HLA engine. It displays the number of logs per minute passing through the MID Server and through the HLA engine.
    Note:
    For the ServiceNow System Logs Retriever integration, the Overview screen doesn’t display the MID Server streaming status because that integration doesn't run on a MID Server.

    The ServiceNow AIOps component shows the total number of alerts that the HLA engine has created. These statistics are updated when the Overview screen loads and are automatically refreshed every minute to show live data. You can change the default auto-refresh time interval through the system property sn_itom_integ_app.overview_page_data_input_stats_auto_refresh_interval_seconds.

    If data streaming fails, the integration is automatically deactivated and the Streaming status marks the component where the failure occurred. In addition, a banner explains the failure and either proposes steps to take to fix it or refers to ServiceNow support.

    Streaming status failure.

    For MID-less or OpenTelemetry Protocol (OTLP) integrations, such as Amazon Data Firehose, the Overview screen displays the ITOM Gateway as a component in the log-processing pipeline. The MID Server component is not shown in the pipeline, because log data is sent directly from the source to the ITOM Gateway. The logs are then processed by the HLA engine to find anomalies. Overview screen displaying the ITOM Gateway component.

    For these integrations, the Overview screen shows the average rate of logs per minute over the last 15 minutes passing through the ITOM Gateway and the HLA engine, similar to the metrics shown for MID-based ingestion.
    Note:
    The ITOM Gateway component is shown only if the MID Server property mid.hla.itom_gateway_streaming.enabled is set to true. For more information, see MID Server properties.
    When the integration isn't streaming live data from the source, a warning message is displayed. There is a 4-hour interval between the last log source time and the current time before this message is generated. You can change the default time interval through the system property sn_itom_integ_app.overview_page_log_source_time_threshold_hours.
    Note:
    The warning message is displayed only when the HLA engine is up and running.

    Log streaming sources

    Depending on your integration, the Log streaming sources table displays the following information about the integration's log streaming sources.
    Note:
    This table isn’t available for the ServiceNow System Logs Retriever integration.
    Table 1. Log streaming sources
    Column Description
    Name The name of the log data source.
    Status The streaming status: Active or Not active.
    Data input The integration streaming the data to your ServiceNow instance.
    MID Server The MID Server to which the log data is streaming.
    Last event time The time when the integration received the latest event.
    Last log processing time The time when the last log was received or processed.
    Raw log lines/sec The average number of raw log lines that streamed to the MID Server per second in the last one-minute interval.
    Note:
    This value represents the number of raw log lines before preprocessing.
    Pre-processed log lines/sec The average number of preprocessed log lines that streamed to the MID Server per second in the last one-minute interval.
    Note:
    This value can differ from the number of raw log lines per second. For example, the difference can be a result of logs having been dropped during preprocessing.

    Example

    As an admin, you use the integration Overview page to verify continuous data flow into the ServiceNow instance. If log streaming fails, HLA doesn't receive the real-time data needed to detect anomalies or generate alerts.

    For example, the Streaming Status for an active Elasticsearch integration may show a red circle with a white X for the MID Server component in the log-processing pipeline. This indicates that logs are not reaching the MID Server.

    The Log Streaming Sources table shows a state of Authentication Failed or Connection Error, with an error message indicating invalid credentials. This points to an issue with the Elasticsearch credentials. Fixing the credentials and either restarting the log service or waiting for the next polling cycle, restores the connection.

    The Log Streaming Sources table now shows a connection state of Connected or Active with successful authentication. The Streaming Status displays a green circle with a white check mark for the MID Server component. With log streaming restored, HLA can resume processing data and generating anomaly alerts.