Configure data inputs (Apache Kafka)

  • Release version: Xanadu
  • Updated August 1, 2024
  • 2 minutes to read

    • Configure a data input for streaming log data from Apache Kafka to your ServiceNow instance. Data input configuration is an essential step in setting up the Health Log Analytics (HLA) application.

    Before you begin

    • Ensure that a MID Server is installed and configured with the Log Ingestion capability enabled. For more information, see MID Server system requirements.
      Important:
      Health Log Analytics does not support IPv6. To work with the application, configure the MID Server to IPv4.
    • If the IP address of the MID Server is exposed by network address translation (NAT), a load balancer or a similar device, it must have a public IP address. In the MID Server properties, add a property named mid.public_ip with the public IP address as the value. For more information, see Create a MID Server property.

    Role required: evt_mgmt_admin

    Procedure

    1. Navigate to All > Health Log Analytics > Data Input > Data Inputs.
    2. On the Data Inputs page, select New.
    3. Choose the Apache Kafka data input.
    4. On the form, fill in the fields.
      For a description of the fields, see Apache Kafka data input configuration fields.
    5. Optional: Select Advanced to set additional configuration fields.
      For a description of the fields, see Apache Kafka data input configuration fields. For information about configuring the advanced settings later, see Configure advanced settings for Apache Kafka data inputs.
    6. On the Query Settings tab, fill in the field.
      For a description of the fields, see Apache Kafka data input configuration fields.
    7. On the Transport tab, fill in the fields.
      For a description of the fields, see Apache Kafka data input configuration fields.
    8. Select Save.
      Health Log Analytics adds the data input record to the Data Inputs table.
    9. Ensure that the data input is configured correctly by selecting Test connection.

      Health Log Analytics tries to connect the MID Server to the data repository.

      If the data input is configured to run on a MID Server cluster, the system tries to connect all the MID Servers contained in the cluster to the repository. The cluster passes the test if at least one of its MID Servers gets connected. This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.

      • If the connection was established, the Test connection button is turned off and the Publish button is enabled.
      • If the connection failed, the reason for the failure displays in the Error message field. Resolve the issue, select Save if you modified the configuration, and then select Test connection to test the connection again.
        Note:
        You can only publish the data input configuration when the connection is created successfully.
      Note:
      You can revert to the last published configuration by selecting Revert Changes. This option is available only when you're modifying a configuration that has been published previously.
    10. Select Publish to publish the data input to the MID Server.

    Result

    The data input configuration process is complete. Health Log Analytics adds the data input record to the Data Inputs table and attaches the configuration file to the data input record. The data input starts streaming log data from Apache Kafka to your ServiceNow instance.

    Note:
    If the HLA engine is down and data has stopped streaming, a notification appears at the top of the data input configuration page. When this happens, contact ServiceNow support.

    What to do next

    Make sure that the data input is streaming data.