Health Log Analytics configuration preferences

  • Release version: Xanadu
  • Updated August 1, 2024
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Health Log Analytics configuration preferences

    This guide details essential configuration preferences for Health Log Analytics, focusing on MID Server settings, log ingestion capabilities, performance tuning, and log retention policies. These configurations ensure efficient log streaming and processing within ServiceNow.

    Show full answer Show less

    Key Configuration for MID Servers

    • Log Ingestion Capability: Must be enabled on the MID Server; enabling all capabilities includes this.
    • Dedicated MID Servers: Recommended to use dedicated MID Servers for log ingestion to optimize performance.
    • Hardware Requirements:
      • Preferred specs: 8 CPUs, 32 GB RAM, network bandwidth up to 10 Gbps, EBS bandwidth up to 4,750 Mbps.
      • Minimum specs: 4 CPUs, 16 GB RAM, Java heap size 8 GB.
    • Java Heap Size: At least 8,192 MB recommended for MID Servers running Health Log Analytics.
    • Java Runtime Environment (JRE): Must be version 11 or above for MID Servers with Health Log Analytics capability in both FIPS and non-FIPS modes.

    Performance and Throughput

    • Expected log ingestion throughput varies by log message size, e.g., up to 20,000 messages per second for 300-byte logs on a Washington DC instance with preferred specs.
    • To increase throughput, adjust ulimit settings or network bandwidth, or reduce log size.
    • Ulimit settings correlate with streaming throughput and differ between in-memory and disk-based queues, affecting maximum rates for various log sizes.
    • From August 2024 release, the Lightning gRPC client can be manually enabled to boost streaming speeds by up to six times, enhancing MID Server to instance communication.
    • By default, each MID Server supports up to 10 data inputs, configurable per server or globally.

    Log Source Retention

    • Default log retention per source is fixed at three days and cannot be changed.
    • Using Health Log Analytics version 22.0.12 or later, available from the ServiceNow Store, customers can modify log retention policies per source or for multiple sources collectively.

    Additional Notes

    • Health Log Analytics requires upgrade to version 34.0.37 by December 2024 to support BC-FIPS version 2.0.
    • Refer to MID Server system requirements for detailed hardware and software prerequisites.

    Commonly used settings for Health Log Analytics properties and general configuration.

    MID Server settings

    • The MID Server log ingestion capability must be enabled.
      Note:
      Enabling All capabilities on the MID Server includes enabling the log ingestion capability.
    • Use dedicated MID Servers for log ingestion whenever possible.
    • To enable MID Servers to run multiple products, Health Log Analytics must have at least the Java Virtual Machine (JVM) memory setting for the standard product for each MID Server thread configuration.
    The preferred MID Server settings for Health Log Analytics are:
      • CPUs: 8
      • RAM: 32 GB
      • Network Bandwidth: Up to 10 Gbps
      • EBS Bandwidth: Up to 4,750 Mbps
      • Maximum Java heap size for MID Server: 8,192 MB
      With the above specifications, the expected log ingestion throughput on a Washington DC instance is as follows:
      • For a log message of 300 bytes: 20,000
      • For a log message of 1.1 KB: 12,300
      • For a log message of 2 KB: 7,970
      The minimum requirements for streaming logs to Health Log Analytics are:
      • CPUs: 4
      • RAM: 16 GB
      • Java heap size for MID Server: 8 GB

      For general information, see: MID Server system requirements.

    • To increase log ingestion throughput, you can either increase the ulimit or the network bandwidth, or decrease the size of the logs being streamed. The ulimit setting can be configured on an individual MID Server. However, the correlation between the ulimit and the throughput can’t be modified.

      The following table lists the ulimit settings for open files relating to network throughput on the MID Server. It shows the size of the logs being streamed from the MID Server to the agent, and the gRPC streaming rate equivalent to the throughput.

      Table 1. Ulimit settings in relation to throughput
      Queue Type Log line size gRPC rate
      In Memory Queue 300 bytes 18,000
      In Memory Queue 1.1 KB 13,000
      In Memory Queue 2 KB 10,000
      Disk-based Queue 300 bytes 11,000
      Disk-based Queue 1.1 KB 5,000
      Disk-based Queue 2 KB 3,000

      Starting from the August 2024 release, you can enhance MID Server communication with the ServiceNow instance by using the Lightning gRPC client, which can increase log streaming speeds to Health Log Analytics by up to six times. The Lightning gRPC client requires manual configuration to activate. For more information, see the Lightning gRPC client - Enabling the new MID gRPC streaming architecture [KB1648419] article in the Now Support Knowledge Base.

    • By default, the number of data inputs per MID Server is limited to 10. You can configure this limitation for an individual MID Server or for all MID Servers.
    • Both in FIPS and non-FIPS mode, MID Servers with Health Log Analytics capability must run on the Java Runtime Environment (JRE) 11 or above.
      Note:
      To support BC-FIPS version 2.0, Health Log Analytics requires an upgrade to version 34.0.37, December 2024.

    Log source retention settings

    By default, log retention per source is set to three days. This setting can't be modified.

    When using Health Log Analytics application, Version 22.0.12 - December 2021 and later, available from the ServiceNow Store , you can modify the log retention policy per source or for multiple sources together. For more information, see Modify the log source retention period.