Stream Connect Message Replication

  • Release version: Yokohama
  • Updated January 30, 2025
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Stream Connect Message Replication

    Stream Connect Message Replication enables ServiceNow customers to replicate data seamlessly between their Apache Kafka environments and ServiceNow instances. It leverages a MID Server with a replicator extension to perform data replication without requiring additional replication services or complex certificate management, as certificates are automatically generated.

    Show full answer Show less

    Configuring and Using Stream Connect Message Replication

    • Enabling MID Server for Replication: The MID Server must be configured to access Hermes endpoints through specific firewall port ranges (Producer: 4000–4050, Consumer1: 4100–4150, Consumer2: 4200–4250). Proxy settings on the MID Server do not affect Hermes connectivity because Hermes uses Kafka-native TCP connections.
    • Kafka Connections and Credentials: You must create Kafka SSL credentials and configure a Kafka connection in ServiceNow. These are then associated via a Connection & Credential alias, which the MID Server uses to connect to Kafka. Hermes connectivity credentials are handled automatically by the MID Server.
    • Creating Message and Topic Replications:
      • A Message Replication record represents a Kafka cluster (one per cluster) and serves as the parent for related topic replications.
      • A Kafka Topic Replication record defines replication from one source topic to one destination topic. One source topic cannot replicate to multiple destinations.
      • These records are stored in the Message Replications and Kafka Topic Replications tables respectively.
    • Monitoring Replications: The system generates metrics records every 60 seconds for each active topic replication, showing details like the message count. Metrics can be viewed in the Message Replication Statistics table or on individual Kafka Topic Replication records.

    Additional Information

    • Required Plugin: Stream Connect Message Replication requires the ServiceNow Stream Connect Installer plugin [com.glide.hub.streamconnect.installer].
    • Roles:
      • messagereplicationadmin: Full control over message replication records and configurations.
      • messagereplicationuser: Read-only access to message replication data.
    • Logging: Most issues are logged in the MID Server logs. Debug logging can be enabled by setting the glide.streamconnect.messagereplication.debug MID Server property to true for detailed troubleshooting.

    Replicate data between your Apache Kafka environment and ServiceNow.

    With Stream Connect Message Replication, you can configure and manage message replications directly from your ServiceNow instance.

    Stream Connect Message Replication uses a MID Server to run the data replications, so you don't need to configure or host additional replication services. It also simplifies the message replication setup by automatically generating the required certificates.

    Enabling a MID Server to replicate data

    Stream Connect Message Replication uses a MID Server with a replicator extension to replicate data to and from your local Kafka. For instructions on how to configure the MID Server, see Configuring MID Servers.

    The MID Server must be able to access Hermes endpoints through the firewall. Work with your network administrator to make sure that the following port ranges are open.
    • Producer: 4000–4050
    • Consumer1: 4100–4150
    • Consumer2: 4200–4250
    Note:
    Proxy settings on the MID Server don't apply to Hermes connectivity. The MID Server proxy configuration applies only to HTTP-based communication. Hermes uses Kafka-native TCP connections, so traffic to the Hermes cluster bypasses proxy settings regardless of how the MID Server is configured.

    Configuring connections and credentials for Kafka

    Stream Connect Message Replication uses a Connection & Credential alias to connect to your local Kafka. To configure a Connection & Credential alias:
    1. Create Kafka SSL credentials with the authentication data required for the connection.
    2. Configure a Kafka connection to connect to your Kafka environment.
    3. Create a Connection & Credential alias, to associate the connection information and credential data. The Connection & Credential alias should have a Connection type of Kafka and must be accessible from the MID Server.

    These steps are for configuring a Connection & Credential alias for connecting to Kafka. You don't need to set up connections or credentials for connecting to Hermes, because the MID Server automatically handles the creation of the required keystore, truststore, and certificates.

    Creating message and topic replications

    Message replication requires Message Replication records and Kafka Topic Replication records.

    A Message Replication record represents a single Kafka cluster. For example, if you have two Kafka clusters, you would create two different Message Replication records, one for each cluster. A Message Replication record is the parent record for all the topics being replicated to or from that cluster. Message Replication records are stored in the Message Replications [sys_sc_message_replication] table.

    A Kafka Topic Replication record specifies the replication from a single source topic to a single destination topic. You can't replicate a single source topic to multiple destinations. You can only replicate to each destination once. Kafka Topic Replication records are stored in the Kafka Topic Replications [sys_kafka_topic_replication] table.

    For a step-by-step guide to creating message and topic replication records, see Create message and Kafka topic replications in Stream Connect.

    Viewing message replication statistics

    Once replications are running, the system creates a metrics record for each active topic replication every 60 seconds. Metrics records provide information about topic replications, including the Message count, which shows the number of messages replicated in each collection interval.

    You can view metrics records on the Message Replication Statistics [sys_sc_channel_replication_metric] table. You can also view metrics records for a particular topic by checking the Message Replication Statistics on its Kafka Topic Replication record.

    For a list of message replication metrics and their descriptions, see Viewing Stream Connect Message Replication statistics.

    Required plugin

    Stream Connect Message Replication requires the ServiceNow Stream Connect Installer [com.glide.hub.stream_connect.installer] plugin.

    Roles

    The message_replication_admin role can create, modify, and delete records in all the message replication tables, including the connection and credential tables, and the message and topic replication tables.

    The message_replication_user role can view records in the message replication tables.

    Logging

    Most issues are logged in the MID Server log. Additional debug logging can be enabled by setting the glide.stream_connect.message_replication.debug MID Server property to true.