Set up data inputs manually

  • Release version: Washingtondc
  • Updated February 1, 2024
  • 3 minutes to read

    • As an alternative to using data input guided setup, you can set up your Health Log Analytics data inputs manually.

    Before you begin

    Important:
    Health Log Analytics does not support IPv6. To work with the application, configure the MID Server to IPv4.
    • You must have an installed and configured MID Server with the log ingestion capability enabled.

      MID Server configuration with Log Ingestion capability enabled.

    • If the MID Server IP address is exposed by network address translation (NAT), a load balancer, or a similar device, it must have a public IP address. In the MID Server properties, add a property named mid.public_ip with the public IP address as the value. For more information, see Create a MID Server property.
    • For information about shipping your logs encrypted using SSL TLS, see the Streaming Data With Rsyslog & Filebeat Using SSL [KB0866319] article in the Now Support Knowledge Base.
    Note:
    Using data input guided setup ensures that you have the minimum required setup for the data input process. For more information, see Set up data inputs using guided setup.

    Role required: evt_mgmt_admin

    Procedure

    1. Configure a data input by performing the procedure in the relevant product documentation.
      Table 1. Data Inputs
      Data Input Description
      Rsyslog or Beats The data input streams log data into your instance using Rsyslog or Beats.
      Splunk The data input streams log data into your instance using Splunk.
      Elasticsearch The data input pulls log data from Elasticsearch indexes into your instance.
      TCP The data input sends raw log messages to your instance directly over a TCP/SSL socket.
      UDP The data input streams raw log messages to your ServiceNow instance directly over a UDP socket.
      GCP Pub/Sub The data input receives log messages that are published to a Google Cloud Pub/Sub topic and streams them to your ServiceNow instance.
      MID Server The data input collects MID Server log files and streams them to your instance.
      Amazon CloudWatch The data input streams log data from Amazon CloudWatch to your ServiceNow instance.
      Amazon S3 The data input streams log data from Amazon S3 (Simple Storage Service) buckets to your ServiceNow instance.
      Microsoft Azure Log Analytics The data input streams log data from Microsoft Azure Log Analytics to your ServiceNow instance.
      Microsoft Azure Event Hubs The data input streams events from Microsoft Azure Event Hubs to your ServiceNow instance.
      Apache Kafka The data input streams log data from Apache Kafka to your ServiceNow instance.
      REST API The data input streams log data to your ServiceNow instance in JSON format.
      Glide Syslog The data input streams log data from the System Log table (Glide Syslog) to the Health Log Analytics AI Engine (Occultus).
      Note:
      Only a single Glide Syslog data input can exist in the system, and only users with the admin role can create and configure it. This data input doesn't run on a MID Server.
      Agent Client Collector The data input streams log messages to your ServiceNow instance using the ServiceNow Agent Client Collector.

      This data input is supported for use with the Agent Client Collector Log Analytics application, available from the ServiceNow Store.
      Note:
      Selecting Test connection at the end of the procedure ensures that your data input is configured correctly. You can only publish a data input configuration when the connection between the MID Server and the data repository has been established.
    2. Identify and address streaming issues to ensure that the data input is streaming log data to the MID Server from all sources.
      For more information, see Identify and resolve log streaming issues.
    3. Determine how Health Log Analytics handles raw log data that is streaming into your instance.
      By default, every incoming log line is auto-mapped to the correct tag. If properties aren't discovered automatically, map the data input sources manually by defining a JavaScript function. For more information, see Map the raw data.
    4. Optional: Modify raw log data before Health Log Analytics maps and structures it.
      For more information, see Edit your raw log data before processing.
    5. Optional: Refine the source type structure to make sure that Health Log Analytics extracts and classifies all properties correctly.
      For more information, see Refine the source type structure.