Schema management in Stream Connect

  • Release version: Yokohama
  • Updated January 30, 2025
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Schema management in Stream Connect

    Schema management in Stream Connect enables ServiceNow customers to import, create, and manage schemas used for serializing and deserializing Kafka messages in the Apache Avro format. Avro schemas reduce payload size and simplify integration with local Kafka instances by converting data between JSON and Avro binary formats. Schemas are stored within ServiceNow, making them accessible to Stream Connect producers and consumers for consistent message formatting.

    Show full answer Show less

    Key Features

    • Schema Import and Creation: Customers can import schemas from the Confluent Schema Registry or create standalone schemas by uploading JSON files or entering JSON strings directly.
    • Schema Storage and Identification: All schemas are stored in the Stream Connect Schemas table, with Confluent and standalone schemas separated into specific tables. Each schema has a unique schema ID—imported from Confluent or generated locally for standalone schemas—which identifies the schema within its registry.
    • Schema Registries: ServiceNow supports two registry types: Confluent Schema Registry and Standalone Schema Registry. Both registries can track schemas in update sets to facilitate moving schemas between environments. By default, tracking is off for Confluent registries (due to schema ID variability) and on for standalone registries.
    • Wire-Level Message Format: Messages follow a standard format for interoperability, starting with a magic byte (0), followed by a 4-byte schema ID, and then the Avro-serialized data payload.
    • Producer and Consumer Configuration: Producers specify a schema to convert JSON payloads automatically to Avro format when sending messages. Consumers specify encoding and schema registry to convert incoming Avro messages back to JSON using the schema ID.
    • Plugin Requirement: Schema management features require the ServiceNow Stream Connect Installer plugin to be activated.

    Practical Benefits

    • Enables efficient Kafka message serialization with reduced payload size and standardized data structures.
    • Supports seamless integration with Kafka producers and consumers by enforcing schema consistency.
    • Facilitates schema lifecycle management within ServiceNow, including environment migration through update sets.
    • Provides flexibility in schema sourcing—either from Confluent registries or custom standalone schemas tailored to specific needs.

    Import and create schemas to send and receive messages in an Apache Avro format. Using an Avro format can reduce the size of the payload and simplify your integration to your local Kafka instance.

    Avro is an open-source data-serialization system that uses schemas to structure encoded data. With an Avro schema, data can be converted from plain-text JSON to an Avro binary format and back. You can store schemas in ServiceNow, so your Stream Connect producers and consumers can use the schemas to serialize Avro messages.

    The following image shows an overview of schema management in Stream Connect. Schemas, stored in schema registries, enable messages in producers and consumers to be converted from plain text to an Avro format and back.

    Diagram showing how Stream Connect uses schemas stored in the schema registries to convert Kafka messages into different formats.

    Schemas

    You can import a schema from the Confluent registry or create your own standalone schema by uploading a JSON file or entering a schema directly as a JSON-formatted string.

    After your schema is imported or created, you can see it on the Stream Connect Schemas [stream_connect_schema] table, which stores both Confluent and standalone schemas. Additionally, Confluent schemas are visible on the Confluent Stream Connect Schema [confluent_stream_connect_schema] table. Standalone schemas are on the Standalone Stream Connect Schema [standalone_stream_connect_schema] table.

    All schemas have a schema ID, a globally unique identifier of the schema. For Confluent schemas, the schema ID is imported from the Confluent registry. For standalone schemas, the schema ID is generated locally and is unique on the instance. By default, the generated schema ID value is the next highest available schema ID on the instance. For example, if your schemas have ID numbers one through five, the next schema you create will have a schema ID of six. You can change the default value.

    Schema IDs are unique per registry. For example, two schemas can both have an ID of one as long as they're in different registries.

    Schema registries

    Every schema belongs to a registry. There are two types of schema registries in ServiceNow: the Confluent Schema Registry and the Standalone Schema Registry.

    Both schema registries have an option to Track in the update set. When this option is enabled, the schemas in that registry are saved to the update set. Saving the schemas to the update set makes it possible to move them from one environment to another. By default, this option is turned off for the Confluent Schema Registry because schema IDs may change from one environment to another. This option is enabled for the Standalone Schema Registry, because if you're creating schemas manually, the schema ID is less likely to change from one environment to another. To change the default setting for either registry, navigate to All > IntegrationHub > Schema Registries, select the registry, and change the Track in the update set option.

    Wire-level message format

    For interoperability, ServiceNow uses a wire-level message format similar to the ones used by other systems. The first byte is set to 0. The next 4 bytes are used for the schema ID. The remaining bytes are used for the data, serialized in an Avro format.
    Byte 0 Magic byte.
    Byte 1–4 Schema ID.
    Remaining bytes Data, serialized in an Avro format.

    Producers and consumers

    Stream Connect producers and consumers can be configured to use an Avro format.

    When configuring a producer, simply specify which schema you want to use. Then when you run the producer, the message payload is generated in JSON and automatically converted to an Avro format using the specified schema. For more information on producers, see the Kafka Producer step or the ProducerV2 API.

    Configuring a consumer is similar. Specify the serialization format as Encoded and select a schema registry. When the consumer receives a message in an Avro format, it's automatically converted to JSON according to the schema for the schema ID received in the message. For more information, see the Kafka Message trigger or the ETL, Transform Map, or Script consumers.

    Plugin

    Schema management features require the ServiceNow Stream Connect Installer [com.glide.hub.stream_connect.installer] plugin.