Deploy and Host LocalAI on Railway

LocalAI is a self-hosted, drop-in replacement for the OpenAI API — one endpoint that handles text generation, image generation, audio transcription, text-to-speech, and embeddings, all running on your own infrastructure. Point any application built on the OpenAI SDK at LocalAI and it works with no code changes. This template deploys the CPU-optimized build with a persistent model store and API-key authentication, so you get a private, multi-modal AI API in one click.

What This Template Deploys

A single service. A persistent volume at /models holds downloaded models — which are large — so they're not re-fetched on every redeploy. The built-in model gallery installs new models from the web UI or an API call.

About Hosting

LocalAI's distinguishing strength is breadth: where most local-AI tools do only chat, LocalAI is a single OpenAI-compatible endpoint for several modalities at once.

One API, many modalities. LocalAI implements OpenAI's endpoints for text generation, image generation (Stable Diffusion), audio transcription (Whisper), text-to-speech, and embeddings. That means one deployment can back a chatbot, generate images, transcribe audio, and produce embeddings for a RAG pipeline — all through the same API key and base URL. For anyone migrating an existing OpenAI-based app, it's a drop-in swap.

Railway is CPU-only, so size models accordingly. There are no GPUs on Railway, and inference runs on CPU. Models under about 4 billion parameters run well here — Phi-4 Mini (3.8B), Gemma 3 1B, TinyLlama, and quantized (Q4_K_M) versions of larger models. They respond more slowly than GPU inference but work reliably within an 8 GB container. Loading an unquantized or oversized model is the common way to hit an out-of-memory crash; stick to quantized models and a modest CONTEXT_SIZE and it's stable.

The /models volume is essential. Model files are typically 500 MB to 4 GB each. Without a persistent volume, every redeploy re-downloads them — slow and wasteful. This template mounts /models so your models persist across restarts and redeploys.

Set THREADS to your core count for best CPU performance, and always set LOCALAI_API_KEY on a public deployment — the endpoint is your AI backend and should never be open. Railway's automatic HTTPS secures it in transit.

Typical cost: ~$5–10/month on Railway for a small model on an 8 GB container, depending on usage. LocalAI is MIT-licensed and free; you pay only for infrastructure.

How It Compares

Ollama is simpler but focuses on LLM chat. vLLM is fast but GPU-oriented and text-only. The OpenAI API does everything but bills per token and keeps your data. LocalAI's niche is being the one self-hosted endpoint that covers text, images, and audio with full OpenAI compatibility — the Swiss Army knife for private, multi-modal AI.

Deploy in Under 5 Minutes

1. Click Deploy on Railway — LocalAI builds automatically using the CPU image (~3 minutes)
2. Set LOCALAI_API_KEY to a strong secret to protect the endpoint
3. Confirm the volume is mounted at /models and set THREADS to your CPU core count
4. Open the web UI and install a small model from the gallery (e.g. Phi-4 Mini or Gemma 3 1B)
5. Point your app's OpenAI base URL at your Railway domain with the API key — no code changes

Existing OpenAI SDK code works by changing only the base URL and key.

Common Use Cases

• Drop-in OpenAI replacement — swap the base URL in an existing app to move off the OpenAI API onto your own infrastructure
• Multi-modal AI backend — one endpoint for chat, image generation, transcription, and embeddings behind a single key
• Private embeddings for RAG — generate embeddings locally to feed a vector store without sending documents to a vendor
• Audio transcription service — Whisper-based speech-to-text on your own server for private audio processing
• Self-hosted image generation — Stable Diffusion image endpoints without a third-party API
• Cost-controlled inference — flat infrastructure instead of per-token billing for steady workloads

Configuration

Size models for CPU and 8 GB RAM. Stick to sub-4B or quantized (Q4_K_M) models and a modest CONTEXT_SIZE. Loading an oversized or unquantized model causes out-of-memory crashes — the most common LocalAI deployment failure.

The /models volume is required. Model files are 500 MB–4 GB each. Without the volume, every redeploy re-downloads them. Always set LOCALAI_API_KEY so the endpoint isn't publicly open.

Dependencies for LocalAI Hosting

• Railway account — expect ~$5–10/month for a small model on an 8 GB container
• A persistent Railway volume at /models (included)
• Adequate RAM for your chosen model (model size + ~2 GB overhead)
• Optional: nothing else — models install from the built-in gallery

Deployment Dependencies

• LocalAI GitHub Repository
• LocalAI Documentation
• LocalAI Model Gallery
• Railway Volumes Documentation

Implementation Details

The template runs localai/localai:latest-cpu on port 8080 with a persistent volume at /models set via LOCALAI_MODELS_PATH. The CPU image is used because Railway provides no GPUs; inference runs on CPU with THREADS controlling parallelism. Models are installed through LocalAI's built-in gallery — from the web UI or an API call — and stored on the volume so they persist across redeploys rather than re-downloading.

LocalAI exposes OpenAI-compatible endpoints for chat completions, image generation, audio transcription and speech, and embeddings, so applications written against the OpenAI SDK connect by changing only the base URL and API key. LOCALAI_API_KEY guards the endpoint, and CONTEXT_SIZE plus quantized model selection keep memory within the container's limits — the main tuning levers on CPU. Model behavior is defined by YAML config files in the models directory, which the gallery installs automatically for curated models.

Because model files are large and CPU inference is memory-sensitive, right-sizing the model to available RAM is the key operational concern; sub-4B and Q4_K_M-quantized models are the reliable choices on an 8 GB Railway container.

Frequently Asked Questions

What can LocalAI do beyond chat? It's a single OpenAI-compatible endpoint for text generation, image generation (Stable Diffusion), audio transcription (Whisper), text-to-speech, and embeddings — one deployment covering multiple modalities, which is its main advantage over LLM-only tools.

Which models run on Railway's CPU? Models under ~4B parameters and quantized versions of larger ones — Phi-4 Mini, Gemma 3 1B, TinyLlama, and Q4_K_M builds. They run on CPU, slower than GPU but reliably within 8 GB.

Why do I need the /models volume? Model files are 500 MB–4 GB. The volume keeps them across redeploys so they aren't re-downloaded each time. It's mounted at /models by default here.

How do I use it as an OpenAI replacement? Change your app's OpenAI base URL to your Railway domain and use LOCALAI_API_KEY as the key. OpenAI SDK code otherwise works unchanged.

Why did my container crash with OOM? Almost always an oversized or unquantized model, or too large a CONTEXT_SIZE. Use a smaller or Q4_K_M-quantized model and reduce the context window.

How do I install models? Through the built-in gallery — the web UI or a single API call — or by setting PRELOAD_MODELS to install at startup. LocalAI auto-detects and loads compatible models from the volume.

Is my data private? Yes. All inference runs on your Railway service; nothing is sent to a third party. That's the point of a self-hosted OpenAI-compatible API.

Why Deploy LocalAI on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying LocalAI on Railway you get a private, multi-modal OpenAI-compatible API in one click — the CPU image configured, a persistent model store, API-key auth, and automatic HTTPS. One endpoint for text, images, audio, and embeddings, drop-in compatible with your existing OpenAI code, on infrastructure you own.