Kafka Toolage: Setup

First written September 2022, last updated September 2026.

We set up four Kafka cluster configurations as a test bed for evaluating tools that monitor and manage Apache Kafka. Here is what the test bed covers and why.

In short

We created a new project for testing Kafka tools; check it out on GitHub at kineticedge/kafka-toolage. It has 4 different Kafka clusters. We spend a lot of time on web searches and trial and error getting configurations working; we hope this saves you some time.

Introduction

When it comes to the configuration of Apache Kafka, the integration can get complicated. The number of configurations of Apache Kafka, Kafka Connect, and Confluent Schema Registry is extensive. When you add in third-party tools, each built by an independent team, it’s challenging to figure out all the configurations.

When you want to evaluate a tool, you want to focus on exploring its features and determining if those features meet your needs; you don't want to worry about setup and especially don’t want to evaluate only to find out you can't easily integrate it with your production environment.

The review we published, Kafka Toolage: Grafana, walks through Grafana against these 4 cluster configurations. This will accelerate your integration time and leave your developers focused on writing Kafka applications and minimizing time spent on infrastructure configuration.

Why Multiple Clusters?

The 4 clusters showcase 5 different Apache Kafka protocols and multiple options for Kafka Connect and Schema Registry.

The non-authenticated cluster is like an internal developer cluster or a POC cluster. It's typically the environment used when trying out Kafka for the first time, or an environment where someone is learning more about Kafka. The cluster demonstrated here has two listeners, PLAINTEXT and SSL, but typically if SSL is desired, the PLAINTEXT listener is omitted.

A SASL authenticated cluster gives the ability to use PLAINTEXT and SSL connections. Why would anyone want to do this? If you have internal IPs only accessible from within the network, specific tools and connections can use zero-copy and improve the performance of inter-broker communication. You also can use PLAIN authentication vs. SCRAM authentication for setting up super-user access. Using SCRAM-based users for admin access is a little more complicated: you have to create those users while ZooKeeper is up (before the brokers ever start), using tooling directly against ZooKeeper. This gives insight into how to do that.

An SSL-authenticated cluster shows how you can use certificates for client authentication. Building up a self-signed certificate process isn't something that is part of most distributions; this project provides one to show how to use openssl to create certificates with the proper X509 extensions.

Finally, an OAUTH-authenticated cluster shows how extensible a tool is, and helps you identify whether a tool supports every authentication mechanism your cluster uses.

All certificates are created internally by this project: a single turnkey generation from CA to broker and client certificates, including intermediate CAs.

The goal here is to verify that the open-source tool being considered will work with your environment. While the clusters presented here don't handle every scenario, they cover many, and we hope that shortens the path to integration with your clusters. We have spent hours on incorrect understandings of configuration, certificates that can be used for client authentication, and the unique configuration settings of many third-party tools.

While reading the documentation is typically the answer, juggling many documents to verify that something meets your needs before spending hours integrating it isn't ideal. This project does that work for you.

Years of Apache Kafka experience are behind this setup. Such setups can easily become 2-4 months of effort within your organization. This project’s goal is to help you get up and running in a fraction of that time.

The Clusters

There are far more scenarios than we cover here. What this provides is a way to test out SASL and SSL authentication, PLAINTEXT and SSL encryption, and basic-auth and no-auth authentication of the RESTful endpoints of Confluent’s Schema Registry and Kafka Connect. Also, a configuration we had yet to try: a custom OAUTH implementation. From a tooling standpoint, we also wanted to see whether a tool can handle multiple Connect clusters associated with a given Kafka cluster. This is why one cluster has two separate Connect clusters: to verify that tools supporting Connect-cluster information can handle more than one (with different Kafka listeners).

Cluster 1: Non-Authenticated Cluster

• Connectivity
• Kafka Brokers
• PLAINTEXT protocol on port 9092
• SSL protocol on port 9093
• Schema Registry
• HTTP
• Kafka Connect
• HTTP
• Configuration
• 1 ZooKeeper
• 3 Brokers
• Inter-Broker Protocol PLAINTEXT
• 1 Schema Registry
• Broker Protocol PLAINTEXT
• Kafka Connect Cluster A
• Broker Protocol PLAINTEXT
• Kafka Connect Cluster B
• Broker Protocol SSL

Cluster 2: SASL Authenticated Cluster

• Connectivity
• Kafka Brokers
• SASL_PLAINTEXT protocol on port 9092 (scram-sha-512 & plain)
• SASL_SSL protocol on port 9093 (scram-sha-512)
• Schema Registry
• HTTPS with Basic Authentication
• Kafka Connect
• HTTPS with Basic Authentication
• Configuration
• 1 ZooKeeper
• 3 Brokers
• Inter-Broker Protocol SASL_PLAINTEXT (plain)
• Schema Registry
• Broker Protocol SASL_PLAINTEXT (plain)
• Kafka Connect
• Two Workers
• Broker Protocol SASL_PLAINTEXT (plain)

Cluster 3: SSL Authenticated Cluster

• Connectivity
• Kafka Brokers
• SSL protocol on port 9093 (SSL Client Authentication required)
• Schema Registry
• HTTPS with Basic Authentication
• Kafka Connect
• HTTPS with Basic Authentication
• Configuration
• 1 ZooKeeper
• 3 Brokers
• Inter-Broker Protocol SSL
• Schema Registry
• Broker Protocol SSL
• Kafka Connect
• Broker Protocol SSL

Cluster 4: OAUTH Authenticated Cluster

• Connectivity
• Kafka Brokers
• SASL_PLAINTEXT protocol on port 9092 (plain)
• SASL_SSL protocol on port 9093 (oauthbearer)
• Configuration
• 1 ZooKeeper
• 3 Brokers
• Inter-Broker Protocol SASL_PLAINTEXT (plain)
• No Schema Registry
• No Connect Cluster
• Open Source Hydra OAuth Server
• Image extended to pre-configure the database and users.
• OAuth Client java library

Demonstration Project

The above clusters and configurations are freely available in an Apache 2.0 Licensed project on GitHub. This licensing doesn't apply to the components being evaluated, and you need to validate and understand their licensing before bringing them into your organization.

Project Notes

• If clusters fail to start, the first thing to check is if you created the certificates. If certificates don’t exist, the brokers fail to start, and the containers that depend on them hang.
• Because multiple clusters run at once, port mapping isn't done. Use the jumphost container to gain access to kafka command line tools to the clusters.
• This project is being developed on a Mac M1 Max with 64GB, with 32GB dedicated to Docker. On a smaller machine, run one cluster at a time.

Classifications

Tool classifications, for the sake of these reviews, are monitoring, observation, and administration.

Example operations for each:

• Monitoring
• Bytes in and out on a given topic
• Disk utilization of brokers
• Number of partitions on a broker
• Connector status and health
• Observation
• Inspect Messages on a topic
• Inspect a Schema
• Consumer Lag
• Administration (Management)
• Move partitions of a topic
• Change the retention time of a topic
• Adjust offsets of a consumer group
• Pause and resume a connector
• Delete a schema

Tools can support multiple operations, but also know that tools are developed with a specific feature set in mind.

Tools

We used this test bed to review Grafana, which uses Prometheus and the Prometheus JMX Exporter. Each review is more than an overview of the features and how they integrate with Apache Kafka. It is also a reference for each cluster integration, in the hope that at least one configuration is close to your setup and makes integration and evaluation quicker.