Local Development for Real-Time Data

First written August 2022, last updated September 2026.

Developers learn Kafka faster when they can break things locally for free. We have heard many stories of developers leaving cloud resources running over the weekend, only to use up too many of their free developer credits. We've also seen developers avoid exploring configurations, nuances, or tools for fear of having to expense unexpected charges.

Project Details

• This real-time development project is available in GitHub as dev-local. All of the Kinetic Edge contributions are Apache License 2.0, but follow the guidelines of each project accordingly, as there are other licenses involved.
• Demonstrations of streaming concepts are provided in dev-local-demos. We hope this toolkit saves you time in building your POC and can improve your understanding of real-time streaming.

Network

A single network is created to be used across each application. The network only needs to be created once, using the network.sh script. Sharing one network across separate docker-compose environments makes it easy to start only what is needed. Components are separated into their own docker-compose and share this network, allowing them to work together to demonstrate real-time processing.

Apache Kafka with Confluent Schema Registry

This is the core component of this toolkit. Four brokers make it easier to understand Kafka: with three brokers and a replication factor of three, it's easy to come away with the misconception that every broker holds a copy of every topic. With four, each partition's replicas land on a subset of the brokers, so stopping one broker visibly affects some partitions and leaves others alone, and you can watch leaders move. That is the mental model people need before they touch a production cluster, and it is hard to get from a diagram.

• 1 ZooKeeper (a KRaft-based variant lives in the repo's kafka-raft folder)
• 4 Brokers
• 1 Confluent Schema Registry

If you don't need to start Schema Registry, a simple way to exclude it is: docker-compose up -d $(docker-compose config --services | grep -v schema-registry). Whether schemas are part of your Kafka deployment is a core decision, which is why Schema Registry is provided here as part of the kafka-core container.

We highly recommend a local installation of Kafka on your laptop for the command-line interface. The Confluent Community Edition is great in that you also get the ksql CLI and the Avro console consumer and producer. Download the community edition from Confluent’s get-started page.

What else is in it

A two-worker Kafka Connect cluster with a ./jars directory for connectors and ./secrets for connector credentials; ksqlDB, which is the quickest way to see stream processing even if Kafka Streams is in your future; Grafana with Prometheus and several open-source Kafka UI tools for monitoring; and containers for Druid, Pinot, Cassandra, Elasticsearch and Kibana, MongoDB, MinIO, MySQL, Postgres, and Oracle (Oracle's licensing means you build that image yourself; the process is documented). Component versions and the exact startup steps are in the README, which is the source of truth for what is current.

Maintenance

• Versions will be updated, and the demonstrations in dev-local-demos will be used to validate those upgrades.
• New applications are added as we use them in client work.