Using Placeholders in Apache Kafka and Amazon Kinesis connections

Using Placeholders in Apache Kafka and Amazon Kinesis connections

When configuring connections to Apache Kafka or Amazon Kinesis, you can use environment variable placeholders to dynamically insert sensitive or environment-specific values (e.g., credentials, file paths, or hostnames). This feature is supported both in the user interface (UI) and through the Public API. Placeholders follow the format ${env:CON_VARIABLE_NAME}, where CON_VARIABLE_NAME is the name of an environment variable available to the application. 

Warning
The environment variable must start with CON_ (e.g., CON_KAFKA_BROKERS). Access to other environment variables is prohibited due to security reasons. Non-prefixed variables (e.g., USER) will not be replaced and will remain as placeholders in the output.

This article covers how to use placeholders when setting up connections, whether through the UI dialog or the Public API.

You can use multiple placeholders in a value, e.g.:
  1. org.apache.kafka.common.security.plain.PlainLoginModule required username="${env:CON_KAFKA_USERNAME}" password="${env:CON_KAFKA_PASSWORD}";
Add the respective environment variables to your container:
  1. export CON_KAFKA_USERNAME="myuser"
  2. export CON_KAFKA_PASSWORD="mypassword123"

Supported Connection Fields

Placeholders can be used in the following connection fields:
  • Addresses: Kafka broker or Kinesis endpoint addresses.
  • Schema Registry: URL for the schema registry.
  • Security Configuration:
    • SASL Configuration
    • Truststore Location 
    • Truststore Password 
    • Keystore Location 
    • Keystore Password 
    • Keystore Private Password 
  • Schema Security Configuration (Kafka only):
    • Schema User Info 
    • Truststore Location 
    • Truststore Password 
    • Keystore Location 
    • Keystore Password 
    • Keystore Private Password 
  • Additional Config Details: Extra configuration settings.
  • Custom Properties: JSON-formatted custom properties.
  • JMX: JSON-formatted JMX configuration.
If an environment variable isn’t found, the placeholder remains unchanged in the field.

    • Related Articles

    • Connect to Kinesis

      In this article, you will learn how to configure Amazon Web Services (AWS) and how to establish a connection from Kadeck to Amazon Kinesis. The article consists of two sections that build on each other: Create a user in Amazon Kinesis Add a ...

    • Connect to Amazon MSK

      This article describes how to connect Kadeck Teams and Kadeck Desktop to Amazon MSK. Important note: Connecting to Amazon MSK with Kadeck Teams is straight forward if Kadeck Teams is deployed inside the same VPC (e.g., as EC2 or ECS instance). Learn ...

    • Deploy Kadeck using Amazon Elastic Container Service (ECS)

      This article describes how to set up a container running Kadeck Teams using Amazon Elastic Container Service (ECS). Please log into your AWS console. A user with permissions to create containers in Amazon ECS is required. 1. Create a task definition ...

    • Connections and Requests

      General Connection Timeout (ms) This applies to kafka and kinesis connections but is also used when Kadeck nodes talk to each other. It's possible that in slow networks and with a lot of data coming from the brokers that 60 secs might not be ...

    • Kafka Connect Rights Management

      To view or manage Kafka Connect connectors or task, the user must have both the KafkaConnectView right and the KafkaConnectManage right. KafkaConnectView and KafkaConnectManage can be assigned for individual connections/environments. Example rights: ...