Stream Connect Message Replication

  • Release version: Xanadu
  • Updated August 1, 2024
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Stream Connect Message Replication

    Stream Connect Message Replication enables ServiceNow customers to replicate data seamlessly between their Apache Kafka environment and their ServiceNow instance. It leverages a MID Server with a replicator extension to manage data replication, eliminating the need for additional replication services. The solution simplifies setup by automatically generating necessary certificates and supports bi-directional message replication with Kafka clusters.

    Show full answer Show less

    Enabling and Configuring MID Server

    The MID Server must be configured with a replicator extension and have network access to Hermes endpoints. Customers need to ensure specific firewall ports are open (Producer: 4000–4050; Consumer1: 4100–4150; Consumer2: 4200–4250) to allow data flow. Detailed MID Server configuration instructions are provided separately.

    Kafka Connection and Credentials Setup

    To connect to Kafka, customers must create Kafka SSL credentials and a Kafka connection within ServiceNow, then associate these via a Connection & Credential alias. This alias must use the Kafka connection type and be accessible from the MID Server. The MID Server automatically handles keystore, truststore, and certificate creation for Hermes connections, so no additional configuration is needed for that.

    Creating Message and Topic Replications

    • Message Replication records represent individual Kafka clusters and act as parent records for topic replications. One record per Kafka cluster is required.
    • Kafka Topic Replication records define replication from one source topic to one destination topic. Replication from a single source topic to multiple destinations is not supported.

    These records are stored respectively in the Message Replications [sysscmessagereplication] and Kafka Topic Replications [syskafkatopicreplication] tables. Customers can follow detailed, step-by-step instructions for creating these records to establish replication relationships.

    Monitoring Replication Performance

    While replications are active, the system generates metrics every 60 seconds for each topic replication. Metrics include the number of messages replicated and other relevant statistics, viewable in the Message Replication Statistics [sysscchannelreplicationmetric] table or directly on Kafka Topic Replication records. This enables customers to track replication health and performance effectively.

    Security and Access Control

    The feature requires the Stream Connect Installer [com.glide.hub.streamconnect.installer] plugin. Role-based access is provided through:

    • messagereplicationadmin role: Allows creation, modification, and deletion of all replication-related records and configurations.
    • messagereplicationuser role: Provides read-only access to replication data.

    Logging and Troubleshooting

    Most operational logs are available in the MID Server log files. For deeper diagnostics, customers can enable debug logging by setting the glide.streamconnect.messagereplication.debug property to true on the MID Server.

    Replicate data between your Apache Kafka environment and ServiceNow.

    With Stream Connect Message Replication, you can configure and manage message replications directly from your ServiceNow instance.

    Stream Connect Message Replication uses a MID Server to run the data replications, so you don't need to configure or host additional replication services. It also simplifies the message replication setup by automatically generating the required certificates.

    Enabling a MID Server to replicate data

    Stream Connect Message Replication uses a MID Server with a replicator extension to replicate data to and from your local Kafka. For instructions on how to configure the MID Server, see Configuring MID Servers.

    The MID Server must be able to access Hermes endpoints through the firewall. Work with your network administrator to make sure that the following port ranges are open.
    • Producer: 4000–4050
    • Consumer1: 4100–4150
    • Consumer2: 4200–4250

    Configuring connections and credentials for Kafka

    Stream Connect Message Replication uses a Connection & Credential alias to connect to your local Kafka. To configure a Connection & Credential alias:
    1. Create Kafka SSL credentials with the authentication data required for the connection.
    2. Configure a Kafka connection to connect to your Kafka environment.
    3. Create a Connection & Credential alias, to associate the connection information and credential data. The Connection & Credential alias should have a Connection type of Kafka and must be accessible from the MID Server.

    These steps are for configuring a Connection & Credential alias for connecting to Kafka. You don't need to set up connections or credentials for connecting to Hermes, because the MID Server automatically handles the creation of the required keystore, truststore, and certificates.

    Creating message and topic replications

    Message replication requires Message Replication records and Kafka Topic Replication records.

    A Message Replication record represents a single Kafka cluster. For example, if you have two Kafka clusters, you would create two different Message Replication records, one for each cluster. A Message Replication record is the parent record for all the topics being replicated to or from that cluster. Message Replication records are stored in the Message Replications [sys_sc_message_replication] table.

    A Kafka Topic Replication record specifies the replication from a single source topic to a single destination topic. You can't replicate a single source topic to multiple destinations. You can only replicate to each destination once. Kafka Topic Replication records are stored in the Kafka Topic Replications [sys_kafka_topic_replication] table.

    For a step-by-step guide to creating message and topic replication records, see Create message and Kafka topic replications in Stream Connect.

    Viewing message replication statistics

    Once replications are running, the system creates a metrics record for each active topic replication every 60 seconds. Metrics records provide information about topic replications, including the Message count, which shows the number of messages replicated in each collection interval.

    You can view metrics records on the Message Replication Statistics [sys_sc_channel_replication_metric] table. You can also view metrics records for a particular topic by checking the Message Replication Statistics on its Kafka Topic Replication record.

    For a list of message replication metrics and their descriptions, see Viewing Stream Connect Message Replication statistics.

    Required plugin

    Stream Connect Message Replication requires the ServiceNow Stream Connect Installer [com.glide.hub.stream_connect.installer] plugin.

    Roles

    The message_replication_admin role can create, modify, and delete records in all the message replication tables, including the connection and credential tables, and the message and topic replication tables.

    The message_replication_user role can view records in the message replication tables.

    Logging

    Most issues are logged in the MID Server log. Additional debug logging can be enabled by setting the glide.stream_connect.message_replication.debug MID Server property to true.