What we learned building koffset with AI

First written February 2026, last updated September 2026.

koffset is our open-source Kafka consumer offset monitoring tool, built from scratch with the goal of replicating metrics from existing lag monitoring tools. We used models to explore algorithms and design, and to help write the code. These are the observations that survived the project. The code is at github.com/kineticedge/koffset.

TL;DR - What we do (now)

• We use a separate model and application for 1:1 chat questions to avoid mixing one-off discussions and tangents with our development.
• We use models to write code that we expect they've seen before.
• For code that is new or less common, we instruct models to argue with us and tell us where we’re wrong.
• We still review all code. We don’t want to put Kinetic Edge’s name on open-source software that hasn’t been written or reviewed by a human.
• We worry about licensing. If an AI model used LGPL code for training, what does that do for our Apache 2.0 licensed projects? What about commercial endeavors?

What we learned

• Using AI reinforces the need for SMEs; it doesn't replace them. This can sound self-serving, but not once did our Kafka knowledge feel unneeded when building this project.
• Days of pair-programming are back, but without knowledge sharing. We gained a great asset in validating our ideas and development. But who has that knowledge once we move on? Even if we save the AI context used to develop the software, will it still be relevant when a new model is released?
• You will need to be less productive in order to get better. Your development process is changing. Give yourself time to adapt.
• Models use old versions of libraries and also mix versions together. Explore multiple models to see which are more current.
• When it comes to Apache Kafka, whose API is evolving rapidly, be ready for a model to give you deprecated code.
• Always give specific versions of software in the context when working with any model. Include more than just version number, such as “use Apache Kafka 4.1 which includes no ZooKeeper (obviously)”.
• When a model gets something wrong, take the time to capture that in a rule to add to your context. If it gets it wrong the first time, it will get it wrong again.
• We were more successful when we used the models to challenge our algorithms and implementations. This aspect of pair-programming was extremely beneficial. Instructing the model to challenge your algorithms gives it context from the start, and uses its training to uncover areas to validate.
• The “praise” the models give you will get old quickly; set up context to minimize it.
• Get unit tests in quickly. It's easy to accept new code from a model, only to get a bug that goes unnoticed, making it difficult to revert.
• Use revision control effectively. For the same reasons as needing unit tests, you want the ability to revert changes.
• Using AI models requires more discipline (not less).

The bug that cost a day

The models we used assumed that getting the latest() offsets for a topic would also provide the max timestamp. It doesn't; it returns -1. We caught it, and we captured the mistake as a rule in our context so the model would stop making it. Then, without disciplined revision control, the same -1 max timestamp came back in later code and cost a day of lost time before we found it again. Two of the habits above came directly from that day: get unit tests in quickly, and keep revision control tight enough that any accepted change can be reverted.