Deploy Kafka [Updated Sep'26]

Deploy and Host Apache Kafka on Railway

Apache Kafka is the distributed event streaming platform behind most modern data pipelines — a self-hosted alternative to Confluent Cloud and Amazon MSK. Publish, subscribe, store and replay streams of records, with consumers reading from any offset rather than draining a queue that forgets. This template runs a single-node broker in KRaft mode on the official image, with logs on a volume and listeners wired for private and external clients — a working bootstrap server, not a broker that accepts one connection then fails.

What This Template Deploys

One process holds both roles, so there is no ZooKeeper service and no separate quorum to size. Everything durable — partition logs, consumer offsets, the metadata log — lives on the volume, the only thing between a redeploy and an empty broker.

About Hosting

Kafka fails on a PaaS in ways that produce no useful error, and almost all of them are decided before the first message is produced.

A regenerated CLUSTER_ID bricks an existing volume. Kafka writes the cluster ID into meta.properties on first format. Supply a different ID on a later deploy and the broker finds the mismatch and refuses to start, reporting an inconsistent cluster ID rather than anything pointing at your config. This template fixes the ID so volume and broker stay in agreement.

KAFKA_ADVERTISED_LISTENERS decides whether clients can use the broker at all. Kafka answers a connection by handing back the address to use afterwards. Advertise the wrong host and clients connect once, get metadata pointing somewhere unreachable, then fail on every produce and fetch. Internal services use the private domain while external clients arrive via TCP proxy on a different host and port — both need advertising.

The log directory must not be the volume mount root. Kafka treats every entry in a log directory as a partition, and Railway creates lost+found at every volume root. Point KAFKA_LOG_DIRS at the mount and the broker aborts on boot. Leave it unset and the image default lands on the ephemeral container layer, where topics vanish on the next deploy without logging anything.

Single node means replication factor 1 — this is not high availability. With one broker there is nowhere to replicate to, so the internal offsets and transaction-state topics must be pinned to 1 or they are never created. A restart interrupts consumers and a lost volume is lost data. Fine for development, staging, internal event buses and low-to-medium production; not for anything needing durability across node failure.

Kafka is a JVM service with a real memory floor. It is not a $5 sidecar. Budget 1–2 GB of RAM before topics get busy, and treat retention as a disk decision: a week of a chatty topic sizes your volume, not your broker.

Typical cost: ~$20–35/month for the broker and a 10 GB volume at light traffic, on rates of $10/GB/month RAM, $20/vCPU/month CPU and $0.15/GB/month volumes. Kafka is Apache 2.0 and free.

How It Compares

The honest edge: if your workload is task dispatch — one worker takes a job, the job disappears — RabbitMQ is simpler and cheaper, and Kafka is overkill. Kafka wins when you need the log itself: many independent consumers on one stream, replay from an arbitrary offset after a fix, or a buffer that never drops events when a consumer restarts. Against Confluent Cloud and MSK the trade is plainer — you take the operations and keep the bill flat.

Deploy in Under 5 Minutes

1. Click Deploy and pick a workspace. The broker comes up with a fixed CLUSTER_ID, a volume and both listeners configured.
2. Wait for the broker to format the log directory on first boot — once, a few seconds.
3. From another Railway service, set bootstrap.servers to the broker's private domain on port 9092 — free, and never leaves Railway.
4. For external clients, enable the TCP proxy under Settings → Networking and use the host and port it generates.
5. Produce and consume a test message on a throwaway topic to confirm both paths work.

Verify before you rely on it: redeploy, then consume that topic from the beginning. If the messages are still there, the volume and cluster ID are both correct.

Common Use Cases

• Decoupling services — publish events instead of calling services directly, so a restarting consumer never takes the producer down.
• Durable event and job logs — keep a replayable history rather than a queue that forgets; reprocess from any offset after a fix.
• Streaming pipelines and CDC — feed change data, clickstreams or telemetry into analytics, search or a warehouse.

Configuration

Never change CLUSTER_ID on a formatted volume. The broker compares it against meta.properties and refuses to start on a mismatch. Recovery means wiping the volume and losing every topic.

Enable the TCP proxy only if you need external clients. A public PLAINTEXT broker has no authentication. Stay on the private network unless you will configure SASL and TLS yourself.

Dependencies for Apache Kafka Hosting

• Railway account — ~$20–35/month for the broker and a 10 GB volume at light traffic.
• Bundled services — none. KRaft mode runs as one process, no ZooKeeper, no external database.
• Volume — required. Holds partition logs, offsets and KRaft metadata; without it every topic is lost on redeploy.
• Optional — a UI such as Kafbat for browsing topics, and a Prometheus exporter for consumer lag.

Deployment Dependencies

• Apache Kafka on GitHub
• Kafka documentation
• KRaft configuration
• Railway volumes

Implementation Details

The broker runs the official apache/kafka image on a pinned tag in combined KRaft mode, holding both roles in one JVM process. The internal listener serves on 9092, the controller on 9093, with the quorum voter list pointing at this node's own private address. Bind addresses leave the host portion empty rather than writing 0.0.0.0, because Railway's private network is IPv6 and a dual-stack wildcard socket is what makes the broker reachable at its .railway.internal name.

Two settings decide whether the deployment survives contact with reality. KAFKA_LOG_DIRS points at a subdirectory inside the volume, not the mount root — clear of the lost+found entry Railway creates, and off the ephemeral layer the image defaults to. CLUSTER_ID is fixed rather than generated, so the value written to meta.properties at first format still matches on every later deploy. Offsets and transaction-state replication are pinned to 1, which one broker can satisfy.

For backups the volume is the whole story — partitions, offsets and metadata all live there. Snapshot it on a schedule if the stream is your system of record, and set retention deliberately: the default keeps a week of every topic, which sizes your disk. Durability across node failure needs more brokers and higher replication factors.

Frequently Asked Questions

Do I need ZooKeeper? No. KRaft replaces it with Kafka's own Raft metadata quorum, and Kafka 4.0 removed ZooKeeper entirely. One process holds both roles here.

How do I connect from another Railway service? Use the broker's private domain on port 9092 as your bootstrap server. Private networking is free and the traffic never leaves Railway.

Can I run this in production? For internal event buses, staging and low-to-medium traffic, yes. One broker means replication factor 1: a restart interrupts consumers and a lost volume is lost data.

Why won't the broker start after a redeploy? Almost always a cluster ID mismatch against meta.properties, or a log directory pointed at the mount root where Kafka finds lost+found and treats it as a malformed partition.

Is Kafka a drop-in replacement for RabbitMQ? No. Kafka is a replayable log; RabbitMQ is a queue that removes messages once consumed. For simple task dispatch RabbitMQ is simpler and cheaper. Pick Kafka when you need replay or many consumers on one stream.

Why Deploy Apache Kafka on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Apache Kafka on Railway you get an event streaming backbone with the traps handled — a fixed cluster ID, a log directory clear of lost+found, listeners advertised correctly for private and external clients, and partitions on a volume that survives redeploys.