Exploring Hermes Messaging Service
Summarize
Summary of Exploring Hermes Messaging Service
The Hermes Messaging Service integrates Apache Kafka with your ServiceNow® instance, facilitating efficient data exchange across business systems. This multi-tenant, multi-cluster service allows your instance to produce and consume large volumes of Kafka events, ensuring streamlined asynchronous data transport with high performance and low latency.
Show less
Key Features
- Event Management: Capture and transmit data events in near real time, organized into topics and partitions for scalability.
- Namespace Organization: Create logical groupings of topics by namespace for access control in domain-separated instances.
- Kafka Clusters: Utilize ServiceNow-hosted Kafka clusters for message delivery management between producers and consumers.
- Diagnostic Tools: Monitor the health and performance of message delivery through Hermes diagnostic tools.
Key Outcomes
By leveraging the Hermes Messaging Service, ServiceNow customers can:
- Publish and process Kafka events at scale, effectively managing spikes in message volume.
- Export data to external applications, ensuring secure log data transfer for reporting and compliance.
- Enhance instance data replication performance through upgraded replication sets.
For further exploration, review topics on configuration, data exchange, and administration of the Hermes Messaging Service.
Learn about the Hermes Messaging Service and how it helps you manage the flow of data between your Kafka environment and your instance.
Hermes Messaging Service overview
Apache Kafka is an open-source data streaming platform that provides a single integration point for exchanging data across business systems in your organization. The Hermes Messaging Service enables you to integrate your Apache Kafka environment with your ServiceNow® instance to exchange data with these business systems.
The Hermes Messaging Service is a multi-tenant, multi-cluster, data transport, and queuing service built on Apache Kafka that enables your instance to produce and consume large volumes of Kafka events.
- Simplify and standardize asynchronous data transport between your Kafka environment and your ServiceNow instance.
- Publish and process Kafka events at scale. Publish events to your Kafka environment from your ServiceNow instance and consume Kafka events from your external systems at a high volume with low latency.
- Effectively process spikes and bursts in message volume to minimize performance impacts on your instance.
- Manage the health and performance of message delivery using Hermes diagnostic tools.
Components
Hermes uses the following components.
- Event
- A record of something that happened, such as the creation of a log entry. Kafka uses events to transmit data between systems in near real time. An event includes a key and a value.
- Topic
- A logical group of events. In Kafka, these events are transmitted as messages. Topics are used to group related events together. Events in a topic are organized using partitions.
- Partition
- A topic is divided into partitions to support scalability and parallel processing. Each event is assigned to a partition based on its key. Events with the same key are always written to the same partition in order of arrival.
- Namespace
- A logical grouping of topics. You can organize topics by namespace to indicate which Kafka cluster they come from. In a domain-separated instance, you can create a namespace for each domain to control access to specific topics.
- Hermes Kafka cluster
- A Kafka cluster hosted in a ServiceNow data center. Hermes uses Kafka clusters to manage the delivery of messages between producers and consumers.
- Producer
- A system or component that creates and sends events to a topic in the Hermes Kafka cluster. The producer sets the event key, which determines the partition where the event is stored.
- Consumer
- A system or component that reads events from one or more topic partitions. Consumers process events in the order they appear within each partition. Consumers can read from multiple partitions simultaneously.
Hermes Messaging Service users
| User | Description |
|---|---|
| Hermes administrator | The Hermes admin defines namespaces for each Kafka cluster or domain and creates topic records in the instance or through the command-line interface (CLI). The Hermes admin monitors topics and views the payload of a message in a Hermes topic using the Hermes Messaging Service topic inspector. |
Hermes Messaging Service workflow
- In a domain-separated instance, the Hermes admin creates namespace records for different domains.
- The Hermes admin creates topic records in the instance or through the CLI.
- The Hermes admin updates or rescans topic records as needed in the instance.
- The Hermes admin monitors topics and views message payloads using the Hermes Messaging Service topic inspector.
Benefits
Different integration points and applications in the ServiceNow platform can use the Hermes Messaging Service to publish and ingest events.
The Hermes Messaging Service is a ServiceNow AI Platform capability that is available as part of Stream Connect, Log Export Service (LES), and Instance Data Replication (IDR).
| Benefit | Application | Users |
|---|---|---|
| Export data from your ServiceNow instance to an external application using the Hermes Messaging Service with Workflow Studio. | Flow Designer | Administrator |
| Export log data securely from your instance to your external data lake for reporting, auditing, or regulatory purposes by using the Hermes Messaging Service with the Log Export Service. | Exploring Log Export Service (LES) | Administrator |
| Trigger flows from incoming Kafka events and trigger outgoing messages to your Kafka environment using the Hermes Messaging Service with Stream Connect. | Using Stream Connect for Apache Kafka | Administrator |
| Improve the performance and processing efficiency of Instance Data Replication (IDR) by upgrading replication sets that you created prior to the Utah release to use the Hermes Messaging Service. | Instance Data Replication | Administrator |
Limitations
- Maximum message size: 2 MB
- Maximum message size per second: 2 MB (produced or consumed)
- Number of topics: No limit
- Maximum number of partitions per topic: 32 partitions
- Maximum number of partitions: 960 (across all external topics and internal topics used by Stream Connect, Log Export Service, and other ServiceNow® apps)
- Retention policy: 36 hours
- Supported compression types for producing messages to Hermes: GZIP, LZ4, and NONE.