Simple Kafka Template

This template provides a straightforward setup for Kafka and Zookeeper, perfect for small projects with minimal configuration.

How to setup

You can connect to Kafka by using the KAFKA_URL environment variable under the "Kafka Broker" service. After connecting, simply create your topics in your code.

NOTE: Ensuring you check if a topic already exists before attempting to create it.

Services

• Kafka Broker: Uses the apache/kafka:latest image.

Kafka Broker Environment Variables

• KAFKA_NODE_ID="1"
  Defines the broker ID for Kafka. It's set to "1" for simplicity. Only change if you have multiple brokers.

• KAFKA_PROCESS_ROLES="broker,controller"
  Specifies the roles for the Kafka process. It's set as both a broker and controller. Leave unchanged unless necessary.

• KAFKA_LISTENERS="PLAINTEXT://:9092,CONTROLLER://:9093"
  Defines the listeners for Kafka. The broker listens on port 9092 and the controller on port 9093. No need to modify these.

• KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://${{RAILWAY_PRIVATE_DOMAIN}}:9092"
  Advertises the Kafka listener on your private domain at port 9092. This should remain as is for the template to work.

To set a public URL:

• Expose the service with a TCP proxy pointing to port 9092.

• This will provide a public url like: random-url.proxy.rlwy.net:12354

• Modify this environment variable to the new public url like this: PLAINTEXT://random-url.proxy.rlwy.net:12354

• KAFKA_CONTROLLER_LISTENER_NAMES="CONTROLLER"
  Specifies the listener names for the controller. Leave this unchanged.

• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT"
  Maps the security protocols for the listeners. Default is PLAINTEXT, which is fine for small projects.

• KAFKA_CONTROLLER_QUORUM_VOTERS="1@${{RAILWAY_PRIVATE_DOMAIN}}:9093"
  Configures the controller quorum voters. Keep this as is unless working with a more complex setup.

• KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR="1"
  Sets the replication factor for the offsets topic. A value of "1" is fine for small-scale use.

• KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR="1"
  Defines the replication factor for transaction state logs. Default of "1" is suitable for basic projects.

• KAFKA_TRANSACTION_STATE_LOG_MIN_ISR="1"
  Sets the minimum in-sync replicas for transaction state logs. Keep it at "1" for simplicity.

• KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS="0"
  Configures the initial rebalance delay for consumer groups. Default is "0", which is appropriate for most cases.

• KAFKA_NUM_PARTITIONS="3"
  Sets the number of partitions for topics. The default is "3", which provides a balanced setup for most small projects.

Note:

It's recommended not to change these environment variables unless you are familiar with Kafka configurations.

By: Bello Shehu Ango
Email: [email protected]
GitHub: github.com/BelloAngo