---
title: "Deploy LocalAI — Private OpenAI API for LLM, Image & Audio"
description: "Self-host one OpenAI-compatible API — LLM, images & audio"
category: "AI/ML"
url: https://railway.com/deploy/localai-multimodal-openai-api
---

# Deploy LocalAI — Private OpenAI API for LLM, Image & Audio

Self-host one OpenAI-compatible API — LLM, images & audio

**[Deploy LocalAI — Private OpenAI API for LLM, Image & Audio on Railway](https://railway.com/template/localai-multimodal-openai-api)**

- **Creator:** SilverBanana
- **Category:** AI/ML

## Template content

### LocalAI https://raw.githubusercontent.com/mudler/LocalAI/master/core/http/static/favicon.svg

- **Image:** localai/localai:latest-cpu
- **Public domain:** Yes

## Documentation

# Deploy and Host LocalAI on Railway

LocalAI is a self-hosted, drop-in replacement for the OpenAI API — one endpoint that handles text generation, image generation, audio transcription, text-to-speech, and embeddings, all running on your own infrastructure. Point any application built on the OpenAI SDK at LocalAI and it works with no code changes. This template deploys the CPU-optimized build with a persistent model store and API-key authentication, so you get a private, multi-modal AI API in one click.

---

## What This Template Deploys

| Service | Purpose |
| --- | --- |
| **LocalAI** | The OpenAI-compatible API, model gallery, and web UI on port `8080` |

A single service. A persistent volume at `/models` holds downloaded models — which are large — so they're not re-fetched on every redeploy. The built-in model gallery installs new models from the web UI or an API call.

---

## About Hosting

LocalAI's distinguishing strength is breadth: where most local-AI tools do only chat, LocalAI is a single OpenAI-compatible endpoint for several modalities at once.

**One API, many modalities.** LocalAI implements OpenAI's endpoints for text generation, image generation (Stable Diffusion), audio transcription (Whisper), text-to-speech, and embeddings. That means one deployment can back a chatbot, generate images, transcribe audio, and produce embeddings for a RAG pipeline — all through the same API key and base URL. For anyone migrating an existing OpenAI-based app, it's a drop-in swap.

**Railway is CPU-only, so size models accordingly.** There are no GPUs on Railway, and inference runs on CPU. Models under about 4 billion parameters run well here — Phi-4 Mini (3.8B), Gemma 3 1B, TinyLlama, and quantized (Q4_K_M) versions of larger models. They respond more slowly than GPU inference but work reliably within an 8 GB container. Loading an unquantized or oversized model is the common way to hit an out-of-memory crash; stick to quantized models and a modest `CONTEXT_SIZE` and it's stable.

**The `/models` volume is essential.** Model files are typically 500 MB to 4 GB each. Without a persistent volume, every redeploy re-downloads them — slow and wasteful. This template mounts `/models` so your models persist across restarts and redeploys.

**Set `THREADS` to your core count** for best CPU performance, and always set `LOCALAI_API_KEY` on a public deployment — the endpoint is your AI backend and should never be open. Railway's automatic HTTPS secures it in transit.

Typical cost: **~$5–10/month** on Railway for a small model on an 8 GB container, depending on usage. LocalAI is MIT-licensed and free; you pay only for infrastructure.

---

## How It Compares

| | LocalAI (this template) | Ollama | OpenAI API | vLLM |
| --- | --- | --- | --- | --- |
| Modalities | Text, image, audio, embeddings | Text (mostly) | All | Text |
| OpenAI-compatible | Full | Partial | Native | Partial |
| Cost model | Flat infra | Flat infra | Per token | Flat infra |
| Data privacy | Fully self-hosted | Self-hosted | Vendor | Self-hosted |
| Runs on CPU | Yes | Yes | N/A | GPU-focused |
| Self-hostable | Yes | Yes | No | Yes |

Ollama is simpler but focuses on LLM chat. vLLM is fast but GPU-oriented and text-only. The OpenAI API does everything but bills per token and keeps your data. LocalAI's niche is being the one self-hosted endpoint that covers text, images, and audio with full OpenAI compatibility — the Swiss Army knife for private, multi-modal AI.

---

## Deploy in Under 5 Minutes

1. Click **Deploy on Railway** — LocalAI builds automatically using the CPU image (~3 minutes)
2. Set `LOCALAI_API_KEY` to a strong secret to protect the endpoint
3. Confirm the volume is mounted at `/models` and set `THREADS` to your CPU core count
4. Open the web UI and install a small model from the gallery (e.g. Phi-4 Mini or Gemma 3 1B)
5. Point your app's OpenAI base URL at your Railway domain with the API key — no code changes

Existing OpenAI SDK code works by changing only the base URL and key.

---

## Common Use Cases

- **Drop-in OpenAI replacement** — swap the base URL in an existing app to move off the OpenAI API onto your own infrastructure
- **Multi-modal AI backend** — one endpoint for chat, image generation, transcription, and embeddings behind a single key
- **Private embeddings for RAG** — generate embeddings locally to feed a vector store without sending documents to a vendor
- **Audio transcription service** — Whisper-based speech-to-text on your own server for private audio processing
- **Self-hosted image generation** — Stable Diffusion image endpoints without a third-party API
- **Cost-controlled inference** — flat infrastructure instead of per-token billing for steady workloads

---

## Configuration

| Variable | Required | Description |
| --- | --- | --- |
| `LOCALAI_API_KEY` | Required | API key protecting the endpoint — set a strong secret, never leave it open |
| `THREADS` | Recommended | CPU threads for inference — set to your core count for best performance |
| `CONTEXT_SIZE` | Recommended | Token context window — keep modest (e.g. `2048`) to avoid OOM on CPU |
| `LOCALAI_MODELS_PATH` | Pre-set | `/models` — the mounted volume where models are stored |
| `PRELOAD_MODELS` | Optional | Models to install at startup, e.g. a gallery model ID |
| `GALLERIES` | Optional | Custom model gallery repositories |
| `F16` | Optional | Half-precision for a memory/quality tradeoff |
| `PORT` | Pre-set | `8080` |

> **Size models for CPU and 8 GB RAM.** Stick to sub-4B or quantized (Q4_K_M) models and a modest `CONTEXT_SIZE`. Loading an oversized or unquantized model causes out-of-memory crashes — the most common LocalAI deployment failure.

> **The `/models` volume is required.** Model files are 500 MB–4 GB each. Without the volume, every redeploy re-downloads them. Always set `LOCALAI_API_KEY` so the endpoint isn't publicly open.

---

## Dependencies for LocalAI Hosting

- Railway account — expect ~$5–10/month for a small model on an 8 GB container
- A persistent Railway volume at `/models` (included)
- Adequate RAM for your chosen model (model size + ~2 GB overhead)
- Optional: nothing else — models install from the built-in gallery

### Deployment Dependencies

- [LocalAI GitHub Repository](https://github.com/mudler/LocalAI)
- [LocalAI Documentation](https://localai.io/)
- [LocalAI Model Gallery](https://localai.io/models/)
- [Railway Volumes Documentation](https://docs.railway.com/volumes)

### Implementation Details

The template runs `localai/localai:latest-cpu` on port `8080` with a persistent volume at `/models` set via `LOCALAI_MODELS_PATH`. The CPU image is used because Railway provides no GPUs; inference runs on CPU with `THREADS` controlling parallelism. Models are installed through LocalAI's built-in gallery — from the web UI or an API call — and stored on the volume so they persist across redeploys rather than re-downloading.

LocalAI exposes OpenAI-compatible endpoints for chat completions, image generation, audio transcription and speech, and embeddings, so applications written against the OpenAI SDK connect by changing only the base URL and API key. `LOCALAI_API_KEY` guards the endpoint, and `CONTEXT_SIZE` plus quantized model selection keep memory within the container's limits — the main tuning levers on CPU. Model behavior is defined by YAML config files in the models directory, which the gallery installs automatically for curated models.

Because model files are large and CPU inference is memory-sensitive, right-sizing the model to available RAM is the key operational concern; sub-4B and Q4_K_M-quantized models are the reliable choices on an 8 GB Railway container.

---

## Frequently Asked Questions

**What can LocalAI do beyond chat?** It's a single OpenAI-compatible endpoint for text generation, image generation (Stable Diffusion), audio transcription (Whisper), text-to-speech, and embeddings — one deployment covering multiple modalities, which is its main advantage over LLM-only tools.

**Which models run on Railway's CPU?** Models under ~4B parameters and quantized versions of larger ones — Phi-4 Mini, Gemma 3 1B, TinyLlama, and Q4_K_M builds. They run on CPU, slower than GPU but reliably within 8 GB.

**Why do I need the `/models` volume?** Model files are 500 MB–4 GB. The volume keeps them across redeploys so they aren't re-downloaded each time. It's mounted at `/models` by default here.

**How do I use it as an OpenAI replacement?** Change your app's OpenAI base URL to your Railway domain and use `LOCALAI_API_KEY` as the key. OpenAI SDK code otherwise works unchanged.

**Why did my container crash with OOM?** Almost always an oversized or unquantized model, or too large a `CONTEXT_SIZE`. Use a smaller or Q4_K_M-quantized model and reduce the context window.

**How do I install models?** Through the built-in gallery — the web UI or a single API call — or by setting `PRELOAD_MODELS` to install at startup. LocalAI auto-detects and loads compatible models from the volume.

**Is my data private?** Yes. All inference runs on your Railway service; nothing is sent to a third party. That's the point of a self-hosted OpenAI-compatible API.

---

## Why Deploy LocalAI on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying LocalAI on Railway you get a private, multi-modal OpenAI-compatible API in one click — the CPU image configured, a persistent model store, API-key auth, and automatic HTTPS. One endpoint for text, images, audio, and embeddings, drop-in compatible with your existing OpenAI code, on infrastructure you own.

## Similar templates

- [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform.
- [stella](https://railway.com/deploy/stella) — Self-host stella with web, API, Postgres, Redis, and object storage.
- [Hermes Agent | OpenClaw Alternative with Dashboard](https://railway.com/deploy/hermes-agent-or-openclaw-alternative-wit) — Self-Hosted Hermes AI Agent for Telegram, Discord & Slack

Open this page in a browser: https://railway.com/deploy/localai-multimodal-openai-api