Set up data inputs for Health Log Analytics manually

  • Release version: Xanadu
  • Updated August 1, 2024
  • 3 minutes to read

    • Set up your data inputs for Health Log Analytics manually. Data input configuration is an essential step in setting up the Health Log Analytics application.

    Before you begin

    Note:
    Consider using the Health Log Analytics data input guided setup, which ensures that you have the minimum required setup for the data input process. For more information, see Set up data inputs using Health Log Analytics guided setup.
    Important:
    Health Log Analytics does not support IPv6. To work with the application, configure the MID Server to IPv4.
    • Ensure that a MID Server is installed and configured with the Log Ingestion capability enabled. For more information, see MID Server system requirements.

      MID Server configuration with Log Ingestion capability enabled.

      Important:
      Health Log Analytics does not support IPv6. To work with the application, configure the MID Server to IPv4.
    • If the MID Server IP address is exposed by network address translation (NAT), a load balancer, or a similar device, it must have a public IP address. In the MID Server properties, add a property named mid.public_ip with the public IP address as the value. For more information, see Create a MID Server property.
    • For shipping your logs encrypted using SSL TLS, see the Streaming Data With Rsyslog & Filebeat Using SSL [KB0866319] article in the Now Support Knowledge Base.

    Role required: evt_mgmt_admin. For the Glide Syslog data input: admin.

    Procedure

    1. Configure a data input by performing the procedure in the relevant product documentation.
      Table 1. Data Inputs
      Data Input Description
      Rsyslog or Beats The data input streams log data into your instance using Rsyslog or Beats.
      Splunk The data input streams log data into your instance using Splunk.
      Elasticsearch The data input pulls log data from Elasticsearch indexes into your instance.
      TCP The data input sends raw log messages to your instance directly over a TCP/SSL socket.
      UDP The data input streams raw log messages to your ServiceNow instance directly over a UDP socket.
      GCP Pub/Sub The data input receives log messages that are published to a Google Cloud Pub/Sub topic and streams them to your ServiceNow instance.
      MID Server The data input collects MID Server log files and streams them to your instance.
      Amazon CloudWatch The data input streams log data from Amazon CloudWatch to your ServiceNow instance.
      Amazon S3 The data input streams log data from Amazon S3 (Simple Storage Service) buckets to your ServiceNow instance.
      Microsoft Azure Log Analytics The data input streams log data from Microsoft Azure Log Analytics to your ServiceNow instance.
      Microsoft Azure Event Hubs The data input streams events from Microsoft Azure Event Hubs to your ServiceNow instance.
      Apache Kafka The data input streams log data from Apache Kafka to your ServiceNow instance.
      REST API The data input streams log data to your ServiceNow instance in JSON format.
      ServiceNow System Logs Retriever The data input streams log data from the ServiceNow System Log table to the Health Log Analytics AI engine.
      Note:
      Only a single ServiceNow System Logs Retriever data input can exist in the system, and only users with the admin role can create and configure it. This data input doesn't run on a MID Server.
      Agent Client Collector The data input streams log messages to your ServiceNow instance using the ServiceNow Agent Client Collector.

      This data input is supported for use with the Agent Client Collector Log Analytics application, available from the ServiceNow Store.
      Note:
      Selecting Test connection at the end of the procedure ensures that your data input is configured correctly. You can only publish a data input configuration when the connection between the MID Server and the data repository has been established.
    2. Identify and address streaming issues to ensure that the data input is streaming log data to the MID Server from all sources.
      For more information, see Identify and resolve log streaming issues.
    3. Determine how Health Log Analytics handles raw log data that is streaming into your instance.
      By default, every incoming log line is auto-mapped to the correct tag. If properties aren't discovered automatically, map the data input sources manually by defining a JavaScript function. For more information, see Map the raw data.
    4. Optional: Modify raw log data before Health Log Analytics maps and structures it.
      For more information, see Edit your raw log data before processing.
    5. Optional: Refine the source type structure to make sure that Health Log Analytics extracts and classifies all properties correctly.
      For more information, see Refine the source type structure.