---
title: "Deploy llama.cpp + Open WebUI"
description: "Deploys llama.cpp (free tokens) +Open WebUI"
category: "AI/ML"
url: https://railway.com/deploy/llamacpp-open-webui
---

# Deploy llama.cpp + Open WebUI

Deploys llama.cpp (free tokens) +Open WebUI

**[Deploy llama.cpp + Open WebUI on Railway](https://railway.com/template/llamacpp-open-webui)**

Machine-readable deploy manifest (JSON, validated by TemplateCI): https://railway.com/deploy/llamacpp-open-webui/manifest.json

- **Creator:** impacte.tech
- **Category:** AI/ML

## Template content

### open-webui

- **Image:** ghcr.io/open-webui/open-webui:main
- **Public domain:** Yes

### llama-server

- **Image:** ghcr.io/ggml-org/llama.cpp:server
- **Start command:** `/app/llama-server -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_0 --host 0.0.0.0 --port 8080 --threads 4 --ctx-size 4096`

## Documentation

# Deploy and Host llama.cpp + Open WebUI

[![Deploy on Railway](https://railway.com/button.svg)](https://railway.com/deploy/Q1VldK)

Run your own private AI chat — **llama.cpp** (`llama-server`) serving open models with raw, first-party performance, and **Open WebUI** on top: a polished, self-hosted ChatGPT-style interface. No API keys, no rate limits, no per-token bills — your prompts never leave your Railway instance.

> 🎯 **Stop renting tokens. Own them.** With closed-model APIs you pay for every token, forever. Here, tokens are unlimited and free — you only pay for infrastructure.

## 🚀 Getting Started with llama.cpp + Open WebUI on Railway | Deployment Guide

Once your Railway deploy is live, the stack downloads your chosen model automatically on first boot and serves it via an OpenAI-compatible API.

### Step 1: Pick Your Model (optional)

The template ships with a sensible small default. To run any open model, set the `MODEL_URL` service variable to a GGUF download URL from Hugging Face, for example:

```
https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf
```

Change the variable and redeploy — the model downloads to the persistent volume and loads. Previously downloaded models stay on the volume, so switching back is instant.

### Step 2: Create Your Admin Account

Open your Open WebUI URL and sign up — the first account becomes the admin.

### Step 3: Chat

Pick the model (auto-discovered from llama-server) and start chatting. Attach documents for RAG, compare responses, or build presets.

### Step 4: Connect Your Apps (Optional)

llama-server exposes a native OpenAI-compatible API. Generate an API key in Open WebUI (Settings → Account → API Keys) and point any OpenAI client at your own endpoint.

## About Hosting llama.cpp + Open WebUI 📖

This template deploys a lean, production-ready two-service stack:

- ⚡ **llama.cpp server** — the inference engine used by virtually every local-AI tool, running as a **single static process** (no daemon/runner layers). Serves the OpenAI-compatible API on the private network, with explicit control over threads, context size, and quantized KV cache.
- 💬 **Open WebUI** — multi-user ChatGPT-style interface: history, markdown, voice, document RAG, web search. The only public-facing service.
- 💾 **Two persistent volumes** — one for GGUF model files, one for users, chats, and uploads. Everything survives redeploys.
- 🔁 **Automatic model bootstrap** — on boot, the start script checks the volume and downloads `MODEL_URL` only if missing.

**Why llama.cpp instead of heavier runtimes:** new model families land here **first** (the engine is updated upstream within days of a release), it runs as one process with no background scheduler, and every performance knob — threads, context, KV cache type, flash attention — is an explicit flag you control.

## Why Deploy llama.cpp + Open WebUI on Railway ✅

One-click deploy with zero server configuration:

- 🟢 No Docker, driver, or build-tool setup — Railway handles it all
- 💾 Persistent volumes keep models, chats, and accounts across deploys
- 🔐 Private networking only — llama-server is never exposed publicly
- 🧵 Predictable performance — pin threads and context to your service size
- 🧩 GPU-capable image variants available (`server-cuda`) where GPUs are offered
- 🔄 Redeploy to update the engine — models and data persist on volumes

## Common Use Cases 💡

- **Private ChatGPT alternative** — personal or team AI chat with zero data leaving your server
- **Free-token API backend** — replace per-token API bills with an OpenAI-compatible endpoint you own
- **Document intelligence** — upload PDFs and docs, chat with them via built-in RAG
- **Team AI workspace** — multi-user accounts, shared prompts, per-user history
- **Open-model experimentation** — test the newest model releases the day they ship, on raw llama.cpp

## Dependencies for llama.cpp + Open WebUI 📦

- **llama.cpp server** — official `ghcr.io/ggml-org/llama.cpp:server` image, GGUF models persisted at `/models` (`server-cuda` variant for GPU deployments)
- **Open WebUI** — official `ghcr.io/open-webui/open-webui` image, data persisted at `/app/backend/data`
- **Railway private networking** — WebUI reaches llama-server via `OPENAI_API_BASE_URL=http://llama-server.railway.internal:8080/v1`

### Deployment Dependencies

- **llama.cpp:** [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) — [GGUF model listings](https://huggingface.co/models?library=gguf&sort=trending)
- **Open WebUI:** [open-webui/open-webui](https://github.com/open-webui/open-webui) — [Docs](https://docs.openwebui.com)

## 🖥️ Minimum Hardware Requirements for llama.cpp + Open WebUI

| Model class | Examples (Q4_K_M GGUF) | Min RAM | Experience |
|---|---|---|---|
| 0.5–1B | Qwen2.5-0.5B, Llama-3.2-1B | 2–4 GB | Snappy on CPU |
| 3–4B | Qwen2.5-3B, Phi-3.5-mini, Gemma-3-4B | 4–8 GB | Snappy on CPU |
| 7–9B | Llama-3.1-8B, Qwen2.5-7B, Mistral-7B | 8–16 GB | Good on CPU, fast on GPU |
| 13–14B | Qwen2.5-14B | 16 GB+ | Usable on CPU |
| 30B+ | Qwen3-32B, DeepSeek-R1-32B | 32 GB+ | GPU recommended |

💡 Start with a 1–4B Q4_K_M model — the sweet spot for quality vs. cost on CPU. Use the `server-cuda` image with `--n-gpu-layers 99` on GPU-enabled services for 10–30× faster inference.

## 💰 Own Your (Free) Tokens — The Economics

| | Closed APIs (per-token) | **This template** |
|---|---|---|
| Token costs | Every request bills | **$0 — unlimited** |
| Heavy month (10M tokens) | $30–150+ | Same infra price |
| Privacy | Prompts leave your control | Prompts stay on your instance |
| Model freedom | Vendor's catalog | Any GGUF on Hugging Face |

**Reality check:** you pay for RAM/CPU while running, not tokens. An always-on 8 GB instance is ~$80/mo — heavy API users break even fast. Everyone else can **scale down when idle**: models reload from the volume in seconds on demand, so light usage costs literal pennies.

## 🐳 Self-Hosting llama.cpp + Open WebUI Outside Railway

Prefer your own hardware? The same stack runs anywhere Docker runs:

```bash
# 1) llama-server with a local GGUF
docker run -d --name llama-server -p 8080:8080 \
  -v ./models:/models \
  ghcr.io/ggml-org/llama.cpp:server \
  llama-server -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 --threads 4 --ctx-size 4096

# 2) Open WebUI, pointed at your server
docker run -d --name open-webui -p 3000:8080 \
  -e OPENAI_API_BASE_URL=http://host.docker.internal:8080/v1 \
  -e OPENAI_API_KEY=local \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main
```

Open `http://localhost:3000`, create your admin account, and chat.

## ❓ FAQ

**How do I change models?** Set `MODEL_URL` to another GGUF URL and redeploy. Each model downloads to the volume once — switching back to a previously downloaded model is instant.

**Why llama.cpp instead of Ollama?** Raw first-party performance, single-process stability, newest model support on day one, and full control over every inference knob. Open WebUI speaks its OpenAI API natively.

**Are my chats and prompts private?** Yes — everything stays on your Railway instance and volumes. Nothing is sent to third parties.

**Can I run multiple models at once?** One model per llama-server process — deploy additional services for more (each with its own volume), and select between them in Open WebUI.

**A model feels slow on CPU.** Use a smaller Q4_K_M quant, pin `--threads` to your CPU quota, or use the `server-cuda` image on a GPU service.

**How do I update the engine?** Redeploy — the image updates, and your models, chats, and accounts persist on the volumes.


## Similar templates

- [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform.
- [stella](https://railway.com/deploy/stella) — Self-host stella with web, API, Postgres, Redis, and object storage.
- [Hermes Agent | OpenClaw Alternative with Dashboard](https://railway.com/deploy/hermes-agent-or-openclaw-alternative-wit) — Self-Hosted Hermes AI Agent for Telegram, Discord & Slack

Open this page in a browser: https://railway.com/deploy/llamacpp-open-webui