Viewing Kafka subscriptions and statistics
Summarize
Summary of Viewing Kafka subscriptions and statistics
This feature enables ServiceNow customers to monitor detailed information about Kafka consumers and their performance. It provides insights into message processing, including the number of records added to topics, processed, and remaining, through Kafka subscriptions created when Kafka streams are activated. Customers can access these details via the IntegrationHub module.
Show less
Key Features
- Kafka Subscription Details: Displays configuration and state of the consumer, including subscription number, consumer ID, topic, state (ACTIVE, REFRESHING, STOPPING), error status, concurrency settings, partition counts, batch size, and auto position policy (earliest or latest message reading).
- Metrics: Provides average input rate (records added), processing rate (records consumed), and queue depth (records remaining) over three intervals—last 5 minutes, last hour, and last 24 hours.
- Partition Groups: Shows ownership status (NONE, PROCESSING, LOCKED), partition ranges, update timestamps, input rates, processing rates, and queue depths for each partition group, helping track message handling and partition allocation.
- Consumer Logs: Access to detailed log entries including timestamps, log levels, messages, and sources. Logging verbosity can be increased by enabling the relevant system property.
- Consumer Statistics: Displays interval-based statistics including produced and consumed messages and bytes, offset sums, lag messages, and partition group details. Interval duration and logging behavior for empty intervals can be configured via system properties.
Practical Use and Benefits
ServiceNow customers can use this capability to efficiently monitor Kafka consumer health and performance, troubleshoot issues through logs, and optimize stream processing configurations. Understanding input rates, processing throughput, and queue depths helps maintain reliable integrations and timely message processing. Configurable properties allow tailoring of log detail and statistics intervals to suit operational needs.
View detailed information about a Kafka consumer and its performance, including the number of records added to the topic, the number of records processed, and the number of records remaining to be processed.
A subscription is a record associated with a consumer. It stores configuration and performance details about the consumer. A subscription is created when a Kafka stream is activated.
- Kafka subscription
- Metrics
- Kafka subscription partition groups
- Consumer logs
- Kafka consumer statistics
To view a subscription record, navigate to . Select the info icon (
) for the subscription you want to view, then select Open Record.
Kafka subscription
| Field | Description |
|---|---|
| Number | Number of the subscription record. |
| State | State of the consumer. Options are:
|
| Topic | Topic the consumer is processing messages from. |
| Consumer ID | Unique identifier for the consumer. |
| Kafka Auto Position Policy | Specifies where the consumer starts reading messages from the topic. Options are:
|
| Has Error | Indicates if the subscription has any errors. |
| Max Concurrency | Maximum number of parallel processors used to consume messages and the number of partition groups. |
| Partition group timeout | Maximum time, in milliseconds, allocated to each partition group in the subscription. |
| Kafka partitions | Number of partitions in the topic. |
| Max Batch Size | Maximum number of messages processed per run. Options are:
|
Metrics
- Topic Input Rate: The average number of records added to the topic per second.
- Consumer Processing Rate: The average number of records processed per second.
- Topic Queue Depth: The average number of records remaining to be processed in the topic.
Kafka subscription partition groups
| Field | Description |
|---|---|
| Number | Number for the partition group. |
| Owner of this partition group | Ownership state of the partition group. Options are:
When a Kafka consumer job is activated, it checks for the earliest updated record with an owner of NONE and picks one. Then the consumer job changes the owner to PROCESSING and processes it for 10 seconds. When it's done processing, the state is changed back to NONE, so it can be picked up and processed again later. If the consumer job processes all the messages, it sets the owner to LOCKED for 10 seconds, after which it's set to NONE again. |
| Updated | Date and time the partition group was last updated. |
| When is the ownership lost | Date and time that ownership is lost. |
| First partition | Number of the first partition for the partition group. |
| Last partition (inclusive) | Number of the last partition for the partition group. |
| Input rate Given for the last five minutes, last hour, and last 24 hours. |
Average number of records added to the partition group per second. |
| Consumer processing Given for the last five minutes, last hour, and last 24 hours. |
Average number of messages consumed from the partition group per second. |
| Queue depth Given for the last five minutes, last hour, and last 24 hours. |
Average number of records remaining to be processed in the partition group. |
Consumer logs
The Consumer Logs tab displays the log entries for the associated consumer. It shows when each entry was created, its log level, the message, and the source. Consumer logs are stored in the Consumer Logs [sys_consumer_log] table.
You can increase the number of log messages added to the consumer logs by enabling the glide.ih.kafka.debug.consume property.
Kafka consumer statistics
| Field | Description |
|---|---|
| Created | Date and time the consumer statistics record was created. |
| Interval | Duration for the consumer statistics record, given in seconds. You can change the length of the interval with the glide.ih.kafka.consumer.log_interval property. The default value is 60 seconds. You can specify whether to log messages to the consumer log when there are no produced or consumed Kafka messages during the interval with the glide.ih.kafka.consumer.log_empty_stats property. The default value is true. |
| Produced messages | Number of messages produced to the topic during the interval time. |
| Consumed messages | Number of messages processed by the consumer during the interval time. |
| Consumed bytes | Number of bytes processed by the consumer during the interval time. |
| Offset | Sum of all the offsets for the partition group. |
| Lag messages | Difference between the end offset and the current offset. |
| End offset | Sum of all the end offsets for the partition group. |
| Partition group | Record for the partition group. |