Configure a Kadeck cluster setup

Configure a Kadeck cluster setup

This article describes how to configure multiple Kadeck instances that run as a cluster.  

Define master nodes

First define a list of master nodes. In this example, we will configure 3 Kadeck instances running as a minimum (to handle 2 simultaneous catastrophic failures):
  1. kadeck-1:2551
  2. kadeck-2:2552
  3. kadeck-3:2553

Configure the nodes

Configure the environment variables for each node as follows.


Node kadeck-1

  1. xeotek_kadeck_cluster_nodes: "kadeck-1:2551,kadeck-2:2552,kadeck-3:2553"
  2. xeotek_kadeck_cluster_hostname: "kadeck-1"
  3. xeotek_kadeck_cluster_port: 2551

Node kadeck-2

  1. xeotek_kadeck_cluster_nodes: "kadeck-1:2551,kadeck-2:2552,kadeck-3:2553"
  2. xeotek_kadeck_cluster_hostname: "kadeck-2"
  3. xeotek_kadeck_cluster_port: 2552

Node kadeck-3

  1. xeotek_kadeck_cluster_nodes: "kadeck-1:2551,kadeck-2:2552,kadeck-3:2553"
  2. xeotek_kadeck_cluster_hostname: "kadeck-3"
  3. xeotek_kadeck_cluster_port: 2553

Auto-scaling instances

Once you have the master nodes set up, you can then have any other amount of auto scaling instances that need just to be configured with the master members list:

  1. xeotek_kadeck_cluster_nodes: "kadeck-1:2551,kadeck-2:2552,kadeck-3:2553"
 
When the instances come online they will register themselves and hand over their own hostname and port to the first master node that is up and running and receive the updated list of members. The current members will then receive an updated list with this new instance in it via gossiping.
 

    • Related Articles

    • Kafka Connect Auto Restart

      Introducing Kafka Connect Auto-Restart in Kadeck. To make life easier for infrastructure and operations teams running Kafka Connect connectors and to ensure higher uptime, it is now possible to have Kadeck automatically restart crashed Kafka Connect ...

    • Horizontal Scaling

      Architecture of a single node To scale horizontally just add servers behind a load balancer (with sticky sessions active). Don't forget to update the cluster configuration Startup Parameters when starting Kadeck on each server.

    • Local cluster

      This article describes how to set up and connect to the local cluster with KaDeck. What is the local cluster functionality? The local cluster, previously known as "embedded cluster", allows you to start up an Apache Kafka broker (and Zookeeper) ...

    • Installations & Environments

      An installation refers to a Kadeck instance or multiple Kadeck instances that share a database (e.g., PostgreSQL) and run in a cluster configuration (with upstream load balancer). Multiple Apache Kafka clusters One Kadeck installation can manage ...

    • Managing Licenses in Kadeck Teams Enterprise

      Introduction In this guide, we'll walk you through the process of assigning licenses to your team members and help you understand how to maintain control over deployment. Confidentiality of Teamid and Secret Upon purchasing Kadeck Teams Enterprise, ...