Managing topics in Hermes

  • Release version: Yokohama
  • Updated January 30, 2025
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Managing topics in Hermes

    This guide explains how to manage topic records within your ServiceNow instance and Kafka topics in the Hermes Kafka cluster using the Hermes Messaging Service. Topics are essential for organizing and storing messages in Kafka, and they are represented as records in the Kafka Topics [syskafkatopic] table in your instance.

    Show full answer Show less

    Key Features

    • Topic Records Management: Topic records link to namespaces and contain fields such as topic name, namespace, and active status. The Full Name field uniquely identifies each topic across namespaces and clusters.
    • Synchronization Job: A scheduled job regularly synchronizes topic records with the Hermes Kafka cluster, creating new topic records, syncing externally created topics, marking inactive topics, and logging unmatched topics.
    • Topic Creation Methods: Topics can be created directly from your ServiceNow instance or externally via Kafka CLI. Including an application ID in the topic name when creating externally ensures proper namespace assignment and avoids synchronization issues.
    • Topic Updates: You can increase the number of partitions in a topic, but cannot reduce them. The namespace assigned to a topic is permanent; moving topics to different namespaces requires deletion and recreation with the new namespace prefix.
    • Topic Deletion: Topics can be deleted either from the Kafka Topics table in your instance (preferred method) or externally via Kafka CLI. External deletion requires manual removal from all Hermes clusters. Topics with active Stream Connect subscriptions cannot be deleted.
    • Rescanning Topics: Rescanning synchronizes topic records by creating records for any topics existing in Hermes but missing in your instance. This runs automatically every 10 minutes or can be triggered manually to expedite synchronization.

    Practical Considerations for ServiceNow Customers

    • Kafka topics are instance-specific and cannot be migrated between instances via update sets; manual recreation is required.
    • Ensure proper namespace management to avoid orphaned topics or synchronization issues, particularly when using external topic creation.
    • Prefer managing topics through the ServiceNow instance interface to maintain synchronization and simplify cluster-wide operations.
    • Regularly monitor topics using the Topic Inspector to oversee the Kafka cluster state and topic health.

    Next Steps

    • Create, update, or delete topics as needed through your ServiceNow instance to maintain synchronization and namespace consistency.
    • Use rescanning to quickly align your topic records with the current state of the Hermes Kafka cluster after external changes.
    • Refer to related procedures on managing namespaces and monitoring topics to optimize your Hermes Kafka cluster management.

    Manage topic records in your instance and Kafka topics in the Hermes Kafka cluster using the Hermes Messaging Service.

    Topic records

    Messages are organized and stored in Kafka topics in the Hermes Kafka cluster. Topic records are stored in the Kafka Topics [sys_kafka_topic] table in your instance. This table shows a list of topics records with a reference to each topic's namespace. Each topic record has a field for the topic's name, related namespace, and whether the topic is active.

    Values in the Name column of the Kafka Topics [sys_kafka_topic] table aren't inherently unique. For example, topic records might have the same Name value when they belong to different namespaces or exist in different clusters. However, values in the Full Name column are always unique.

    A scheduled job runs regularly to synchronize topic records with the topics found in Hermes. This job performs the following functions:
    • Creates topic records for any discovered topics, assigning them to the appropriate namespace based on the prefix for the topic name.
    • Synchronizes topics created from external sources, such as the Kafka command-line interface (CLI). For example, when a topic is created from an external source on the near cluster, the job runs and creates the topic on the far cluster as well.
    • Logs a warning message if a topic can't be matched to any namespace. If a topic can't be matched to any namespace, the topic record isn't created. There's a default, prefix-less namespace called the Default Namespace that matches all topics. The only time a topic doesn't match any namespace is when the Default Namespace has been deleted or modified.
    • Marks topics as inactive if no Kafka topic was found for them.

    You can monitor and view topics in the Hermes Kafka cluster using the topic inspector. For more information, see Monitoring topics in the Hermes Kafka cluster.

    Kafka topics are specific to a particular instance. This means you can't migrate a topic record from the Kafka Topics [sys_kafka_topic] table to another instance using an update set. Instead, you must manually create the topic in the target instance.

    Creating topics

    You can create topics in the Hermes Kafka cluster using either of the following methods:
    • Create the topic from your instance. This creates a record the Kafka Topics [sys_kafka_topic] table and creates a Kafka topic in the Hermes Kafka cluster. See Create a topic in Hermes.
    • Create the topic externally. For example, create the topic using the Kafka command-line interface (CLI).
      Note:
      To avoid syncing issues between the cluster and your instance, include the application ID when creating topics externally. For example, without the application ID, a topic created from the CLI with a name like snc.instancename.fulfillment.topicname is created in the default namespace instead of the fulfillment namespace. To ensure the topic is created in the fulfillment namespace, include the application ID in the name using the format snc.instancename.fulfillment.app_id.topicname. Replace app_id with the application ID of your choice.

    If, for some reason, a topic can't be created in the Hermes Kafka cluster, it won't be created in the Kafka Topics [sys_kafka_topic] table either.

    Updating topics

    If necessary, you can change the number of partitions in a topic. Note that you can only increase the number of partitions in a topic. For more information, see Update a topic in Hermes.

    After it's created, the namespace assigned to a topic record is permanent. If you create a namespace record and want to move existing topics to it, you must delete the topics and recreate them in the Hermes Kafka cluster with the new namespace prefix. If the topics weren't created through the instance, new topic records will be created and assigned to the matching namespace record after a rescan.

    Deleting topics

    You can delete topics from the Hermes Kafka cluster using either of the following methods:
    • Delete the topic directly from the Kafka Topics [sys_kafka_topic] table in your instance. This method automatically deletes the topic from all the Hermes Kafka clusters in one action. See Delete a topic in Hermes
    • Delete the topic externally. For example, delete the topic using the Kafka command-line interface (CLI).
      Important:
      Deleting the topic externally requires that you delete the topic from both Hermes clusters manually by specifying each set of ports (410x and 420x). If possible, delete the topic through the instance instead.

    You can't delete a topic if it has subscriptions in Stream Connect.

    Rescanning topics

    Rescanning creates topic records for any topics in the Hermes Kafka cluster that don't already have corresponding topic records. For example, if you create topics in Hermes using the CLI, you can rescan to create corresponding topic records in the Kafka Topics [sys_kafka_topic] table immediately instead of waiting for the scheduled job to create them.

    • The rescan occurs automatically every ten minutes.
    • Created topic records are assigned to existing namespaces based on prefix matching rules.
    Sometimes topic records in your instance don't match what's in Hermes. This situation can happen in the following cases:
    • When the ServiceNow Integration Hub Kafka Consumer (com.glide.hub.kafka_consumer) plugin is enabled after there are already topics in Hermes.
    • When topic records are deleted accidentally.
    • When namespace definitions are changed, and you want to assign existing topics to the new namespaces.