Apache Kafka integration configuration fields

  • Release version: Xanadu
  • Updated April 7, 2025
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Apache Kafka integration configuration fields

    This guide details the configuration fields available for setting up the Apache Kafka integration within the Health Log Analytics application on ServiceNow, version 26.0.17 and later. The integration enables you to pull log data from Kafka clusters into your ServiceNow instance for analysis and monitoring.

    Show full answer Show less

    Key Configuration Sections

    Integration Setup

    • Integration Name: Assign a unique name to identify the Kafka integration instance.
    • Execute on: Choose whether to use a specific MID Server or a MID Server cluster to run the data input.
    • MID Server name / Cluster: Select the MID Server or failover cluster that will pull the Kafka log data. Only MID Servers supporting basic authentication (not mTLS) are supported. Log ingestion is automatically enabled if not already activated on the selected MID Server(s).
    • Service Instance: Bind the Kafka log data to a specific ServiceNow service instance.
    • Data Source: This is fixed to Kafka and is read-only.
    • Description: Optionally add descriptive text to help identify the integration.

    Data Retrieval Configuration

    • Kafka Node Names: Provide a comma-separated list of Kafka broker host and port pairs (HOST:PORT) to connect to the Kafka cluster. It is not necessary to include all servers.
    • Topics: Specify the Kafka topics to subscribe to for log data, as a comma-separated list.
    • Kafka Credentials: Select or create Kafka SSL credentials to authenticate to the Kafka brokers securely.
    • Group ID: Set the Kafka consumer group name for managing the consumption state.

    Advanced Settings

    • Timeout: Milliseconds to wait during polling when no data is available (default 500 ms).
    • Default Timezone: Time zone applied to events lacking time zone information; defaults to GMT but is configurable.
    • Sub sample Receive/Drop Ratios: Settings to control event batching and reduce the volume of received or fetched events.
    • Character Encoding: UTF-8 encoding is used by default and is read-only.
    • Node Discovery Timeout: Timeout in milliseconds for discovering Kafka nodes (default 30 ms).
    • Max Length in Bytes: Maximum event size allowed (default 32766 bytes).
    • Drop if Queue is Full: Option to discard logs during MID Server load; default is false (do not drop).

    Practical Implications for ServiceNow Customers

    By correctly configuring these fields, you enable efficient and reliable ingestion of Kafka log data into ServiceNow Health Log Analytics. The use of MID Server clusters with failover ensures high availability and resilience. Understanding and setting advanced options allows you to tune performance and event volume according to your environment’s needs. This configuration supports basic authentication only and requires automatic or manual enabling of log ingestion on MID Servers.

    Description of the fields on the Apache Kafka integration configuration forms for Health Log Analytics.

    Table 1. Provide details
    Field Description
    Integration Name Unique name of this integration. For example: My Kafka integration. This field is required.
    Note:
    When you fill in this field, the generic name displayed on the form adjusts automatically to match the name you entered.
    Execute on Option to determine whether to use a specific MID Server or a MID Server cluster.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.
    MID server name

    (Only when the Execute on field is set to Specific MID Server)

    MID Server to which log data from Apache Kafka is pulled. This field is required.
    Note:
    • You can select only MID Servers that support basic authentication. MID Servers that support mTLS are not listed.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. You can modify the maximum number by adding the property sn.occ.log_ingestion.max_datainputs_per_mid to the MID Server and then changing the default value.
    • If log ingestion is not enabled for the selected MID Server, Health Log Analytics enables it automatically.
    MID Server Cluster

    (Only when the Execute on field is set to Specific MID Server Cluster)

    The MID Server cluster to which the log data is pulled.

    The data input runs on a single MID Server in the cluster until that MID Server fails. The system then moves all the data input tasks to the next available MID Server in the cluster according to the configured order.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.

    Note:
    • Health Log Analytics supports only failover MID Server clusters. In these clusters, multiple MID Servers are grouped together for failover protection. When selecting a cluster from the data input form, the MID Server Clusters list displays only failover clusters.
    • The MID Server cluster must include only MID Servers that support basic authentication. mTLS is not supported for log ingestion.
    • Log ingestion must be enabled for each MID Server in the cluster. If log ingestion is not enabled for the active MID Server, Health Log Analytics enables it automatically.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. A cluster passes capacity validation if it contains at least one MID Server with fewer than 10 data inputs running on it, even when that MID Server is down.
    For more information about MID Server clusters, see Configure a MID Server cluster.

    This field is required.
    Service instance The service instance (formerly the application service) to which to bind the log data. This field is required.
    Data source The source of the log data that the integration pulls to your ServiceNow instance: Kafka. This field is read-only.
    Description Option to add a brief description of the integration to help identify it.
    Table 2. Set data retrieval method
    Field Description
    Kafka node names A comma-separated list in the format HOST:PORT,HOST:PORT. This field is required.

    The list does not have to include all the Apache Kafka Cluster servers.
    Topics A comma-separated list of topics to which the data input must subscribe. This field is required.
    Kafka credentials The Apache Kafka credentials.

    You can select existing Kafka SSL credentials, or create new ones by selecting Create Kafka credentials from the drop-down list. For a description of the fields on the Kafka SSL credentials form, see Kafka SSL credentials fields.
    Group ID The name of the Apache Kafka Consumer Group.
    Table 3. Advanced settings
    Field Description Default value
    Timeout The time, in milliseconds, spent waiting in the poll if data is not available in the topics. 500
    Default timezone The time zone of events that the system will use if a log does not specify the time zone.

    By default, the system uses GMT in such cases, but you can specify a different time zone.

    		 GMT
    Sub sample receive ratio The number of events to batch together, out of which all but one will be discarded. This setting is used to decrease the number of received events. -1
    Character encoding The character encoding for this data input. This field is read-only. UTF-8
    Node discovery timeout The time, in milliseconds, before node discovery times out. 30
    Sub sample drop ratio The number of events to batch together, out of which one will be discarded. This setting is used to reduce the number of fetched events. -1
    Max length in bytes The maximum length, in bytes, of events. 32766
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server. False