Managing topics in Hermes

  • Release version: Xanadu
  • Updated August 1, 2024
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Managing topics in Hermes

    Hermes Messaging Service allows you to manage Kafka topics both within your ServiceNow instance and in the Hermes Kafka cluster. Topics are represented as records in the Kafka Topics [syskafkatopic] table in your instance, linked to namespaces that help organize and differentiate topics.

    Show full answer Show less

    A scheduled synchronization job keeps topic records aligned with actual Kafka topics in the Hermes cluster, handling creation, updates, and status changes. Kafka topics are instance-specific and cannot be migrated via update sets; manual creation is required when moving between instances.

    Managing Topics

    • Creating Topics: You can create topics either from the ServiceNow instance (which adds a record and a Kafka topic) or externally using Kafka CLI. When creating externally, including the application ID in the topic name ensures proper namespace assignment and prevents synchronization issues.
    • Updating Topics: You can increase the number of partitions for existing topics, but the assigned namespace is permanent. Changing a topic’s namespace requires deleting and recreating the topic with the new namespace prefix.
    • Deleting Topics: Topics can be deleted from the instance table, which removes them from all Hermes Kafka clusters in one action, or externally via Kafka CLI on both clusters. Deletion is blocked if the topic has active Stream Connect subscriptions. Deleting via the instance is recommended for ease and consistency.
    • Rescanning Topics: Rescanning synchronizes topic records with existing Kafka topics, creating records for any missing topics based on namespace prefix matching. This can be triggered manually or occurs automatically every ten minutes to maintain consistency.

    Important Details for ServiceNow Customers

    • Topic names in the Kafka Topics table are not unique alone; uniqueness is ensured by the full name including the namespace.
    • If a topic cannot be matched to a namespace, it will not be created unless the Default Namespace exists; deleting or modifying the Default Namespace can cause unmatched topics.
    • Topics created externally without the proper naming convention (including application ID) may default to the Default Namespace, causing synchronization issues.
    • Topics cannot be migrated between instances using update sets; manual creation is necessary.
    • Always prefer deleting topics through the instance to ensure deletion from all clusters and to avoid manual multi-cluster cleanup.

    Related Operations

    • Create a topic in Hermes: Add topics from within ServiceNow to manage Kafka topics directly.
    • Update a topic in Hermes: Modify topic records to adjust configurations such as partition count.
    • Delete a topic in Hermes: Remove topics and their messages safely from both the instance and Kafka clusters.
    • Rescan topics: Synchronize your instance’s topic records with the actual Kafka topics to maintain consistency.

    Manage topic records in your instance and Kafka topics in the Hermes Kafka cluster using the Hermes Messaging Service.

    Topic records

    Messages are organized and stored in Kafka topics in the Hermes Kafka cluster. Topic records are stored in the Kafka Topics [sys_kafka_topic] table in your instance. This table shows a list of topics records with a reference to each topic's namespace. Each topic record has a field for the topic's name, related namespace, and whether the topic is active.

    Values in the Name column of the Kafka Topics [sys_kafka_topic] table aren't inherently unique. For example, topic records might have the same Name value when they belong to different namespaces or exist in different clusters. However, values in the Full Name column are always unique.

    A scheduled job runs regularly to synchronize topic records with the topics found in Hermes. This job performs the following functions:
    • Creates topic records for any discovered topics, assigning them to the appropriate namespace based on the prefix for the topic name.
    • Synchronizes topics created from external sources, such as the Kafka command-line interface (CLI). For example, when a topic is created from an external source on the near cluster, the job runs and creates the topic on the far cluster as well.
    • Logs a warning message if a topic can't be matched to any namespace. If a topic can't be matched to any namespace, the topic record isn't created. There's a default, prefix-less namespace called the Default Namespace that matches all topics. The only time a topic doesn't match any namespace is when the Default Namespace has been deleted or modified.
    • Marks topics as inactive if no Kafka topic was found for them.

    You can monitor and view topics in the Hermes Kafka cluster using the topic inspector. For more information, see Monitoring topics in the Hermes Kafka cluster.

    Kafka topics are specific to a particular instance. This means you can't migrate a topic record from the Kafka Topics [sys_kafka_topic] table to another instance using an update set. Instead, you must manually create the topic in the target instance.

    Creating topics

    You can create topics in the Hermes Kafka cluster using either of the following methods:
    • Create the topic from your instance. This creates a record the Kafka Topics [sys_kafka_topic] table and creates a Kafka topic in the Hermes Kafka cluster. See Create a topic in Hermes.
    • Create the topic externally. For example, create the topic using the Kafka command-line interface (CLI).
      Note:
      To avoid syncing issues between the cluster and your instance, include the application ID when creating topics externally. For example, without the application ID, a topic created from the CLI with a name like snc.instancename.fulfillment.topicname is created in the default namespace instead of the fulfillment namespace. To ensure the topic is created in the fulfillment namespace, include the application ID in the name using the format snc.instancename.fulfillment.app_id.topicname. Replace app_id with the application ID of your choice.

    If, for some reason, a topic can't be created in the Hermes Kafka cluster, it won't be created in the Kafka Topics [sys_kafka_topic] table either.

    Updating topics

    If necessary, you can change the number of partitions in a topic. Note that you can only increase the number of partitions in a topic. For more information, see Update a topic in Hermes.

    After it's created, the namespace assigned to a topic record is permanent. If you create a namespace record and want to move existing topics to it, you must delete the topics and recreate them in the Hermes Kafka cluster with the new namespace prefix. If the topics weren't created through the instance, new topic records will be created and assigned to the matching namespace record after a rescan.

    Deleting topics

    You can delete topics from the Hermes Kafka cluster using either of the following methods:
    • Delete the topic directly from the Kafka Topics [sys_kafka_topic] table in your instance. This method automatically deletes the topic from all the Hermes Kafka clusters in one action. See Delete a topic in Hermes
    • Delete the topic externally. For example, delete the topic using the Kafka command-line interface (CLI).
      Important:
      Deleting the topic externally requires that you delete the topic from both Hermes clusters manually by specifying each set of ports (410x and 420x). If possible, delete the topic through the instance instead.

    You can't delete a topic if it has subscriptions in Stream Connect.

    Rescanning topics

    Rescanning creates topic records for any topics in the Hermes Kafka cluster that don't already have corresponding topic records. For example, if you create topics in Hermes using the CLI, you can rescan to create corresponding topic records in the Kafka Topics [sys_kafka_topic] table immediately instead of waiting for the scheduled job to create them.

    • The rescan occurs automatically every ten minutes.
    • Created topic records are assigned to existing namespaces based on prefix matching rules.
    Sometimes topic records in your instance don't match what's in Hermes. This situation can happen in the following cases:
    • When the ServiceNow Integration Hub Kafka Consumer (com.glide.hub.kafka_consumer) plugin is enabled after there are already topics in Hermes.
    • When topic records are deleted accidentally.
    • When namespace definitions are changed, and you want to assign existing topics to the new namespaces.