Skip to content
Kinetic Edge
Back to Insights
Data EngineeringBy KE Engineering Team

Kafka Configuration from Environment Variables

Kafka Configuration from Environment VariablesDATA ENGINEERING cover for Kafka Configuration from Environment VariablesENVCLIENT PROPERTIESKAFKA_BOOTSTRAP_SERVERSbootstrap.serversKAFKA_SECURITY_PROTOCOLsecurity.protocolKAFKA_GROUP_IDgroup.idNO CONFIG FILE IN THE IMAGEDATA ENGINEERINGKafka Configurationfrom EnvironmentVariables// ENV VARS · CONTAINER READY

First written September 2023, last updated September 2026.

Many applications use environment variables for configuration, especially when they are deployed within a container. With just a little bit of code, you can use the same behavior for your Java Kafka clients.

Introduction

The Kafka configuration convention makes it easy to use environment variables for configuration. The convention is that all properties are lowercase and all separation is with periods; properties can be pulled safely from the environment.

Overrides

  • Loop through all environment variables, filtering out only those with a prefix, such as “KAFKA_CLIENT_”.
  • Replace all “_” characters with “.” and lowercase everything. If _ ever becomes part of a client property name, you'll need an escape (for example __ for _); the code below leaves that out.
  • Wrap System.getenv() so tests can swap in a map.
java
private static Map<String, String> environment() {  return System.getenv();} public static Map<String, String> environmentProperties(String prefix) {  return environment().entrySet().stream()      .filter(e -> e.getKey().startsWith(prefix))      .map(e -> {        String key = e.getKey().substring(prefix.length()).replace("_", ".").toLowerCase();        return Map.entry(key, e.getValue());      })      .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));}

At the heart of it, that’s it. But it isn’t quite that simple.

Connection (& Secrets)

The handling of secrets typically requires more restrictions within an organization. We prefer to load all connection settings from a shared-secret mounted in the container. Put all connection settings in this file, e.g. bootstrap.servers, security.protocol, and sasl.mechanism. Any settings or secrets that change based on where the application is deployed become part of container initialization, not application configuration.

java
public static Map<String, Object> load(String propertyFile) {  try {    File file = new File(propertyFile);    if (file.exists() && file.isFile()) {      Properties properties = new Properties();      try (InputStream is = new FileInputStream(file)){        properties.load(is);        return new HashMap<>(properties.entrySet().stream()            .collect(Collectors.toMap(e->e.getKey().toString(),Map.Entry::getValue)));      }    } else {      throw new RuntimeException("unable to read property file.");    }  } catch (IOException e) {    throw new RuntimeException(e);  }}

Here, connection information is extracted from a property file; there are more secure ways of pulling that data into the container.

Putting it all Together

Build the config in four layers. Each layer overwrites the one before it, so the order is the policy: environment variables can't override the mounted connection secrets, and nothing can override the immutables.

java
private Map<String, Object> properties() {   Map<String, Object> map = new HashMap<>();   // 1. Defaults: what the development team knows is right for this client  map.putAll(Map.ofEntries(        Map.entry(ProducerConfig.BATCH_SIZE_CONFIG, 200_000L),        Map.entry(ProducerConfig.LINGER_MS_CONFIG, 200L),        Map.entry(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy")  ));   // 2. Overrides: anything operators set in the environment  map.putAll(KafkaUtil.environmentProperties("KAFKA_CLIENT_"));   // 3. Connection properties (secrets): mounted into the container, win over the environment  map.putAll(PropertiesUtil.load("/secrets/connection.properties"));   // 4. Immutables: the application decides these, and nothing overrides them  map.putAll(Map.ofEntries(        Map.entry(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()),        Map.entry(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName())  ));   return map;}

Who controls what

Operators get environment overrides, security keeps secrets in mounted files, and the app keeps control of what it must.

Working on something like this?

Start a Conversation
Back to Insights