Deploy and Host LocalAI on Railway

LocalAI is the free, open-source drop-in replacement for the OpenAI API: it serves /v1/chat/completions, /v1/embeddings, /v1/audio/* and /v1/images/* from models that run on your own CPU, with no API key sold by anybody and no request leaving your project. This template runs LocalAI v4 as a single service with one volume, an inference backend and a chat model already installed in the image, and an API key enforced from the first request.

About Hosting LocalAI

LocalAI v4 ships a deliberately small base image. The inference backends live in a separate OCI gallery and the models live in a model gallery, so a stock container starts with neither: the API answers backend not found: llama-cpp, and the deployer has to install a backend and a model by hand before anything works. That is a reasonable design for a workstation and a poor one for a platform deploy, where the first thing a person does with the URL is send a request to it.

Two further things need handling on Railway. A volume is a bind mount, so it hides whatever the image build wrote underneath its mount path — anything baked into the image has to be kept outside the mount and copied across on the first boot. And LocalAI does not read $PORT: it takes its listen address from LOCALAI_ADDRESS, while Railway's health check dials the port it injected, so the address has to be derived from the injected value before the server starts.

This image handles all three, and refuses to start without an API key — LocalAI serves both inference and its web UI to anyone when no key is set, and the URL is public from the moment Railway assigns the domain.

Why Deploy LocalAI on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying LocalAI on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

Concretely, for this app: inference is a long-running, memory-resident process with a disk attached, which is exactly what Railway's always-on services and volumes provide, and the public domain plus TLS mean any OpenAI SDK can point at it by changing one base URL.

Common Use Cases

Give an app, an agent framework or a coding tool an OpenAI-compatible endpoint whose per-token cost is a fixed monthly container instead of a metered bill.
Keep prompts and documents inside your own infrastructure, where no third-party provider sees them.
Serve embeddings, transcription or image generation from the same endpoint by installing further models from the built-in gallery.

Dependencies for LocalAI Hosting

A Railway volume, mounted at /data.

Deployment Dependencies

LocalAI documentation: https://localai.io/
Upstream image: https://quay.io/repository/go-skynet/local-ai
Wrapper image source: https://github.com/bon5co/localai-railway

What this template does that a stock deploy does not

It answers on the first request. The cpu-llama-cpp backend and the Apache-2.0 qwen3-1.7b chat model are installed at build time, and a warmup call loads the model once the server reports ready — so the first caller gets tokens, not a model download.
The baked assets survive the volume. Every LocalAI storage path is relocated under /data, and pristine copies kept outside the mount are seeded onto it on the first boot only, so a redeploy neither loses the model nor re-copies it.
Nobody else can use your endpoint. LOCALAI_API_KEY is generated per deploy and the container refuses to start if it is empty. Anonymous requests return 401.
$PORT is honoured, by exporting LOCALAI_ADDRESS from the injected port, so the deployment answers at its own domain and the health check passes.
Threads are sized to the container. llama.cpp otherwise sizes its pool from the host's core count, which a container always over-reports; this image reads the quota from /sys/fs/cgroup/cpu.max.
The image is pinned to LocalAI v4.9.0 and prebuilt.

After deploying

Point any OpenAI client at the deployment URL and use the generated LOCALAI_API_KEY from the service's variables:

curl $URL/v1/chat/completions \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-1.7b","messages":[{"role":"user","content":"hello"}]}'

Install further models from the gallery with POST /models/apply, or browse them in the web UI at the same URL. Larger models want more memory: raise the service's plan before pulling one.