Deploy Simple Kafka

Simple Kafka Setup For Small Projects

Deploy Simple Kafka

Kafka Broker

apache/kafka:latest

Just deployed

Simple Kafka Template

This template provides a straightforward setup for Kafka (with internal and public connectivity), perfect for small projects and lightweight production deployments.

How to Setup

You can connect to Kafka using one of the two provided environment variables:

  • KAFKA_URL → for internal access within your private or production network
  • KAFKA_PUBLIC_URL → for external access (via TCP proxy) from outside the network

In your application, simply use the appropriate environment variable based on your deployment environment.

Example:

# Internal app (same Railway network)
KAFKA_URL=kafka-broker.railway.internal:9092

# External app (internet access)
KAFKA_PUBLIC_URL=random-url.proxy.rlwy.net:12354

After connecting, create your topics in your code — but always check if a topic already exists before creating it.

Services

  • Kafka Broker: Uses the apache/kafka:latest image. Runs both broker and controller roles in one container for simplicity.

Kafka Broker Environment Variables

KAFKA_NODE_ID="1"
KAFKA_PROCESS_ROLES="broker,controller"
KAFKA_LISTENERS="PLAINTEXT://:9092,PLAINTEXT_HOST://:29092,CONTROLLER://:9093"
KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://${{KAFKA_URL}},PLAINTEXT_HOST://${{KAFKA_PUBLIC_URL}}"
KAFKA_CONTROLLER_LISTENER_NAMES="CONTROLLER"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
KAFKA_CONTROLLER_QUORUM_VOTERS="1@${{RAILWAY_PRIVATE_DOMAIN}}:9093"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR="1"
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR="1"
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR="1"
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS="0"
KAFKA_NUM_PARTITIONS="3"
KAFKA_URL="${{RAILWAY_PRIVATE_DOMAIN}}:9092"
KAFKA_PUBLIC_URL="${{RAILWAY_TCP_PROXY_DOMAIN}}:${{RAILWAY_TCP_PROXY_PORT}}"

Explanation of Key Variables

VariableDescription
KAFKA_URLInternal Kafka endpoint used for private/production connections within the same network.
KAFKA_PUBLIC_URLPublic TCP proxy endpoint used for external connections (e.g., from local dev or monitoring tools).
KAFKA_ADVERTISED_LISTENERSAdvertises both internal (PLAINTEXT://${KAFKA_URL}) and external (PLAINTEXT_HOST://${KAFKA_PUBLIC_URL}) listeners so Kafka knows how to respond to different clients.
KAFKA_LISTENERSDefines the listener ports inside the container: internal (9092), external (29092), and controller (9093).
KAFKA_LISTENER_SECURITY_PROTOCOL_MAPMaps each listener to its security protocol — all use PLAINTEXT for simplicity.
KAFKA_CONTROLLER_QUORUM_VOTERSSets the controller configuration for single-broker operation.
KAFKA_NUM_PARTITIONSDefault partition count for new topics (3 is a balanced default).

Notes

  • Internal services on the same network should always use KAFKA_URL.
  • External clients (like staging or developer laptops) should use KAFKA_PUBLIC_URL.
  • All security protocols are PLAINTEXT — not recommended for sensitive data or production-grade clusters without TLS/SASL.

By: Bello Shehu Ango 📧 Email: [email protected] 💻 GitHub: github.com/BelloAngo


Would you like me to add a connection test section (e.g., using kafka-console-producer and kafka-console-consumer examples) to verify both internal and public connections? It’s handy for quick debugging.


Template Content

More templates in this category

View Template
smoothmq
A drop-in replacement for AWS SQS

View Template
Redpanda+Console : 1$
Enterprise-Grade Redpanda Streaming Platform

View Template
Maildev
Ephemeral SMTP server for development purposes.