Stream Connect quick start guide

  • Release version: Xanadu
  • Updated August 1, 2024
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Stream Connect quick start guide

    This guide provides ServiceNow customers with a concise introduction to setting up and using Stream Connect to exchange data between their ServiceNow instance and an Apache Kafka environment. Stream Connect facilitates data integration via six key components—two producers and four consumers—that connect through the Hermes Messaging Service to a Kafka environment, enabling seamless data flow both to and from ServiceNow.

    Show full answer Show less

    Key Features

    • Producers:
      • Kafka Producer step in Workflow Studio – Allows sending messages from ServiceNow workflows.
      • ProducerV2 API – Enables programmatic publishing of messages to Kafka.
    • Consumers:
      • Kafka Message trigger in Workflow Studio – Retrieves Kafka messages for workflow processing.
      • Script consumer – Provides scripted retrieval of Kafka messages.
      • Extract Transform Load (ETL) consumer – Imports data using ETL definitions for scheduled or event-driven loads.
      • Transform Map consumer – Uses transform maps for data transformation during import.
    • Hermes Messaging Service: Acts as the communication layer linking ServiceNow and the Kafka environment securely.
    • Secure connection setup: Utilizes mutual authentication via a keystore and truststore to securely connect to Hermes through the ADCv2 gateway.

    Getting Started and Configuration

    The guide outlines five main steps: setting up and testing a secure connection to Hermes, followed by configuring and using each producer and consumer. It leverages Apache Kafka command-line tools to represent the Kafka environment for testing and configuration, compatible with multiple Kafka and OpenSSL versions.

    To establish a secure connection, customers must create certificates to generate keystore and truststore files, then verify connectivity through the ADCv2 gateway. Afterward, customers can configure consumers (Kafka Message trigger, Script consumer, ETL consumer, Transform Map consumer) to retrieve data and producers (Kafka Producer step, ProducerV2 API) to publish data.

    Practical Application

    • ServiceNow customers can use Workflow Studio triggers and steps or APIs to integrate real-time Kafka messaging in workflows.
    • ETL and Transform Map consumers enable sophisticated data transformation and import capabilities from Kafka events into ServiceNow tables.
    • Secure, authenticated connections ensure reliable and safe data exchange between ServiceNow and Kafka environments.

    This quick start guide gives you an overview of how to set up and use each of the six Stream Connect producers and consumers to exchange data between your ServiceNow instance and a Kafka environment.

    Stream Connect has two producers that send data from ServiceNow to a Kafka environment.
    • Kafka Producer step in Workflow Studio
    • ProducerV2 API
    Stream Connect also has four consumers that retrieve data from a Kafka environment.
    • Kafka Message trigger in Workflow Studio
    • Extract Transform Load (ETL) consumer
    • Transform Map consumer
    • Script consumer

    To link your ServiceNow instance to a Kafka environment, Stream Connect uses the Hermes Messaging Service. The following diagram shows each of the producers and consumers and how they connect to your Kafka environment, shown here as the customer site, through Hermes.

    Figure 1. Stream Connect producers and consumers
    Diagram showing Stream Connect producers and consumers and how they connect to a Kafka environment, described as the "customer site", through the Hermes Messaging Service.

    For more information, see Using Stream Connect for Apache Kafka and the Hermes Messaging Service.

    Getting started

    To represent the customer site, this guide uses Apache Kafka command-line tools run in your computer. The configuration parameters used with this tool can be used to configure any client connection to the Application Delivery Controller, version 2 (ADCv2) gateway using the Kafka protocol.

    All the given commands are tested with OpenSSL version (LibreSSL 2.8.3) and Apache Kafka binary distribution version kafka_2.13-3.4.0.tgz. However, all the commands should work with any version you have in your local environment.

    Quick start steps

    This quick start guide has five steps. Steps 1 and 2 cover how to set up and test a connection to Hermes. Steps 3 through 5 show you how to configure and use each of the Stream Connect producers and consumers to send and receive data.

    1. The ADCv2 gateway uses mutual authentication to authenticate requests, so the first step is to create a keystore and truststore to use with your favorite Kafka client. The following page shows you how to create the required certificates: Set up a secure connection to the Hermes Messaging Service.
    2. After you set up a secure connection to Hermes, verify the keystore and truststore have been correctly generated by Testing the connection to Hermes through the ADCv2 gateway.
    3. Now you can Use the Kafka Message trigger and Script consumer to consume messages. The first part of this page shows you how to use the Kafka Message trigger in Workflow Studio to retrieve messages from Kafka. The second part shows you how to use the Script consumer to retrieve messages.
    4. And Use the ETL and Transform Map consumers to import data. ETL definitions and transform maps specify the transformation logic to use when pulling data through scheduled imports. You can use the same ETL definitions and transform maps to transform the events received through Kafka.
      Note:
      To use the ETL consumer, you need a robust import set transformer. Likewise, to use the Transform Map consumer, you need a transform map.
    5. Finally, you can Use the Kafka Producer step and the ProducerV2 API to publish messages. This page shows you how to use both the Kafka Producer step in Workflow Studio and the ProducerV2 API to push messages to Kafka.