Data Generator

The Datagen connector can be used as a source to generate sample data based on a specified data.type. Use the Datagen connector when you want to test pipeline SQL statements using random data without setting up connections to external systems.

Overview

Connector name

datagen

Type

source

Delivery guarantee

at most once

Use the Datagen Connector

Follow these steps to use the Datagen Connector.

  1. From the Connections page, select the Datagen Connector and complete the following fields.

    UI Field Property Name in the Decodable CLI Description

    Data Type

    data.type

    The type of sample data that you want this connection to emit. Must be one of the following:

    - envoy: Emits records formatted in Envoy’s default layout pattern. See Default format pattern in the Envoy documentation for an example of what the emitted record looks like.

    - counter: Emits records with counter and time fields. The counter field is a monotonically increasing bigintand the time field is a timestamp encoded as a string.

    Delay

    delay

    Specifies the delay in milliseconds to wait between records. Must be greater than 100.

    Defaults to 100.

    Value Format

    value.format

    Must be set to JSON.

  2. Select the stream that you’d like to connect to this connector. Then, select Next.

  3. On the "Define this Connection’s schema" page, select Next. You cannot modify the schema for the Datagen connector.

  4. Give the newly created connection a Name and Description and select Save.