Producing and consuming messages from a Kafka client

  • Release version: Australia
  • Updated March 12, 2026
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Producing and consuming messages from a Kafka client

    This guide explains how ServiceNow customers can produce and consume messages to and from Hermes using a Kafka client. Hermes employs a dual Kafka cluster setup for failover, requiring specific configuration steps for reliable message exchange.

    Show full answer Show less

    Key Configuration Requirements

    • Topic Creation: Begin by creating a topic in the Hermes Kafka cluster to enable message exchange.
    • Processes Needed: Set up three processes:
      • One producer process to send messages to Hermes.
      • Two consumer processes to receive messages, corresponding to each Kafka cluster for failover.
    • Consumer Clients: Configure two distinct consumer clients, each with separate bootstrap addresses and port ranges (4100-4150 for Consumer1 and 4200-4250 for Consumer2). Each client can have multiple consumer group IDs.

    Network and Connectivity

    • Port Ranges: Ensure these port ranges are open on your network firewall before communication:
      • Producer: 4000-4050
      • Consumer1: 4100-4150
      • Consumer2: 4200-4250
    • No Proxy Support: Hermes uses Kafka-native TCP connections, so HTTP proxy settings on MID Servers do not apply for Kafka traffic. Proxy configurations affect only HTTP-based communication.
    • Port Accessibility Testing: Use openssl sclient commands against your instance and specified ports to verify connectivity through firewalls.

    Bootstrap Addresses

    • Producers: Connect to Hermes Kafka clusters on ports 4000-4050 using bootstrap address strings like:
      <instancename>.service-now.com:4000,<instancename>.service-now.com:4001,...
    • Consumers: Configure two separate bootstrap address strings for the two consumer clients:
      • Consumer A uses ports 4100-4150.
      • Consumer B uses ports 4200-4250.
    • Maintain this pattern to ensure connectivity and failover support as additional clusters are integrated.

    Using the Command-Line Interface (CLI)

    • Listing topics may show topics from one or both Kafka clusters, depending on synchronization timing.
    • Topics created for failover have a three-letter cluster prefix to differentiate them.
    • Internal topics will appear alongside user-created topics.
    • Topics cannot be deleted from both clusters with a single CLI command; instead, delete topics via the ServiceNow instance interface.

    Learn how to produce and consume messages to Hermes from a Kafka client.

    To begin exchanging messages from a Kafka client, start by creating a topic in the Hermes Kafka cluster. After you successfully create the topic, you must create a total of three processes to communicate with Hermes.

    • One process is required to produce messages to Hermes.
    • Two processes are required to consume messages from Hermes. Two processes are required because Hermes uses a pair of Kafka clusters for failover purposes. If one cluster goes down, data is produced to the other Hermes Kafka cluster.
      Important:
      You must configure two distinct consumer bootstrap addresses, one for each consumer client.

    One consumer process uses ports in the 410x range and the other uses ports in the 420x range. This means you must define two distinct clients, one for each port range. Within each client configuration, you can define consumer group IDs.

    Required port ranges

    Work with your network administrator to confirm that the following port ranges are open before you begin exchanging messages using Hermes:
    • Producer: 4000-4050
    • Consumer1: 4100-4150
    • Consumer2: 4200-4250
    Note:
    Proxy usage is not supported for Hermes connectivity. Hermes uses Kafka-native TCP connections (not HTTP). Therefore, proxy settings configured on a MID Server are ignored and don't provide a supported path to reach the Hermes cluster. The MID Server proxy configuration applies only to HTTP-based communication and does not apply to Kafka traffic. This applies to all applications that connect to Hermes, including Log Export Service and Stream Connect.

    To confirm that the ports are accessible through the firewall, run the following command:

    openssl s_client -connect <instance_name>.service-now.com:<port> -servername <instance_name>.service-now.com
    • Change the service-now.com domain if your instance uses a different domain name.
    • Complete the test using ports from each of the three port ranges. For example: test 4000-4003, 4100-4103, and 4200-4203.

    If the command fails internally but succeeds from a client outside the network, consult your network administrator to gain access through the network firewall.

    Bootstrap addresses

    Use the following port mappings to connect producers and consumers to the Kafka cluster bootstrap addresses. All Application Delivery Controllers used by Hermes follow this same convention.

    Important:
    The following Hermes bootstrap address examples use ports 4000-4003, 4100-4103, and 4200-4203. However, you must open the following port ranges to ensure connectivity as additional clusters are added to the Hermes Messaging Service.
    • Producer: 4000-4050
    • Consumer1: 4100-4150
    • Consumer2: 4200-4250

    Producer clients use ports ranging from 4000 to 4050. For example:

    Producers = "<instance_name>.service-now.com:4000,<instance_name>.service-now.com:4001,<instance_name>.service-now.com:4002,<instance_name>.service-now.com:4003"

    Because Hermes uses a pair of Kafka clusters, you must configure two consumer clients with separate consumer bootstrap addresses.

    Important:
    You must configure two distinct consumer bootstrap addresses, one for each consumer client.
    • The first consumer client uses ports ranging from 4100 to 4150. For example:
      Consumer A = "<instance_name>.service-now.com:4100,<instance_name>.service-now.com:4101,<instance_name>.service-now.com:4102,<instance_name>.service-now.com:4103"
    • The second consumer client uses ports ranging from 4200 to 4250. For example:
      Consumer B = "<instance_name>.service-now.com:4200,<instance_name>.service-now.com:4201,<instance_name>.service-now.com:4202,<instance_name>.service-now.com:4203"

    When you configure producer and consumer properties for your own Kafka client, use this string pattern.

    Using the command-line interface (CLI)

    When using the list command to view a list of topics, a list of current topics are returned from one or both clusters. Topics might be returned from only one cluster, depending on when the last synchronization occurred. Topics created for failover purposes are differentiated by a three-letter cluster identification prefix.

    When accessing the Hermes clusters using the CLI, internal topics appear in addition to any topics you've created. For details, see KB1705399.

    You can't delete a topic from both Kafka clusters using a single command in the command-line interface (CLI). However, you can delete the topic from both clusters by deleting the topic record in your instance. See Delete a topic in Hermes.