How To: Set up SASL Authentication with Kafka

A common method for securing client connections to Kafka brokers is through SASL (ref).

Decodable supports a number of SASL authentication mechanisms, both with and without SSL/TLS encryption.

Setup Decodable Kafka Connection

We’ll assume here that you already have a Decodable account and have gotten started with the Decodable CLI. If you haven’t done that yet, see The Decodable CLI to learn how to install and setup the Decodable CLI.

Create a Stream

decodable stream create --name kafka_sasl_in            \
  --description "input stream"                          \
  --field value=string

Create a Connection

When configuring a SASL connection, the required properties are:

security-protocol (one of SASL_SSL or SASL_PLAINTEXT)
sasl.mechanism (one of PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512)
sasl.username (the provided username or API key)
sasl.password (the provided password or API secret key)

🚧 Self-Signed Certificates

Note that when using SASL authentication with SSL/TLS encryption (SASL_SSL), the broker must be configured to use a trusted certificate. Self-signed certificates are not supported at this time.

These can be used as Decodable connection properties, e.g.

decodable connection create --connector kafka --type source          \
  --name kafka-sasl-source                                           \
  --description "Kafka source connection with SASL/SSL"              \
  --stream-id=<stream-id>                                            \
  --field value=STRING                                               \
  --prop value.format=raw                                            \
  --prop bootstrap.servers=<broker_list>                             \
  --prop topic=source_raw                                            \
  --prop security.protocol=SASL_SSL                                  \
  --prop sasl.mechanism=PLAIN                                        \
  --prop sasl.username="<username>"                                  \
  --prop sasl.password="<password>"

Test the Connection

The quickest way to test the connection is to activate it and run a preview job. After activation, we can verify that the connection is activated successfully by checking the actual state.

Note that it may take up to 1 minute for the state to update.

Activate the Connection

decodable connection activate <connection_id>

decodable connection get <connection_id>
#sample output
kafka-sasl-source
  id                       a3ead34d
  description
  connector                kafka
  type                     source
  stream id                abbcfccd
  schema
    0  value                 STRING
  properties
    bootstrap.servers        <broker_list>
    value.format             raw
    sasl.mechanism           PLAIN
    sasl.username            <username>
    security.protocol        SASL_SSL
    topic                    source_raw
  target state             RUNNING
  actual state             RUNNING
  create time              2021-11-23T15:15:02Z
  update time              2021-11-23T15:15:02Z

Create a Preview Job

Run a preview to read from the stream the source Kafka connection writes into. If you produce raw strings to the source_raw topic, you should see sample data coming out from the preview command output.

Note that it may take up to 1 minute for the data to show up.

decodable pipeline preview "SELECT * FROM kafka_sasl_in"