Deploy llama.cpp + Open WebUI

Deploy and Host llama.cpp + Open WebUI

Run your own private AI chat — llama.cpp (llama-server) serving open models with raw, first-party performance, and Open WebUI on top: a polished, self-hosted ChatGPT-style interface. No API keys, no rate limits, no per-token bills — your prompts never leave your Railway instance.

🎯 Stop renting tokens. Own them. With closed-model APIs you pay for every token, forever. Here, tokens are unlimited and free — you only pay for infrastructure.

🚀 Getting Started with llama.cpp + Open WebUI on Railway | Deployment Guide

Once your Railway deploy is live, the stack downloads your chosen model automatically on first boot and serves it via an OpenAI-compatible API.

Step 1: Pick Your Model (optional)

The template ships with a sensible small default. To run any open model, set the MODEL_URL service variable to a GGUF download URL from Hugging Face, for example:

https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf

Change the variable and redeploy — the model downloads to the persistent volume and loads. Previously downloaded models stay on the volume, so switching back is instant.

Step 2: Create Your Admin Account

Open your Open WebUI URL and sign up — the first account becomes the admin.

Step 3: Chat

Pick the model (auto-discovered from llama-server) and start chatting. Attach documents for RAG, compare responses, or build presets.

Step 4: Connect Your Apps (Optional)

llama-server exposes a native OpenAI-compatible API. Generate an API key in Open WebUI (Settings → Account → API Keys) and point any OpenAI client at your own endpoint.

About Hosting llama.cpp + Open WebUI 📖

This template deploys a lean, production-ready two-service stack:

• ⚡ llama.cpp server — the inference engine used by virtually every local-AI tool, running as a single static process (no daemon/runner layers). Serves the OpenAI-compatible API on the private network, with explicit control over threads, context size, and quantized KV cache.
• 💬 Open WebUI — multi-user ChatGPT-style interface: history, markdown, voice, document RAG, web search. The only public-facing service.
• 💾 Two persistent volumes — one for GGUF model files, one for users, chats, and uploads. Everything survives redeploys.
• 🔁 Automatic model bootstrap — on boot, the start script checks the volume and downloads MODEL_URL only if missing.

Why llama.cpp instead of heavier runtimes: new model families land here first (the engine is updated upstream within days of a release), it runs as one process with no background scheduler, and every performance knob — threads, context, KV cache type, flash attention — is an explicit flag you control.

Why Deploy llama.cpp + Open WebUI on Railway ✅

One-click deploy with zero server configuration:

• 🟢 No Docker, driver, or build-tool setup — Railway handles it all
• 💾 Persistent volumes keep models, chats, and accounts across deploys
• 🔐 Private networking only — llama-server is never exposed publicly
• 🧵 Predictable performance — pin threads and context to your service size
• 🧩 GPU-capable image variants available (server-cuda) where GPUs are offered
• 🔄 Redeploy to update the engine — models and data persist on volumes

Common Use Cases 💡

• Private ChatGPT alternative — personal or team AI chat with zero data leaving your server
• Free-token API backend — replace per-token API bills with an OpenAI-compatible endpoint you own
• Document intelligence — upload PDFs and docs, chat with them via built-in RAG
• Team AI workspace — multi-user accounts, shared prompts, per-user history
• Open-model experimentation — test the newest model releases the day they ship, on raw llama.cpp

Dependencies for llama.cpp + Open WebUI 📦

• llama.cpp server — official ghcr.io/ggml-org/llama.cpp:server image, GGUF models persisted at /models (server-cuda variant for GPU deployments)
• Open WebUI — official ghcr.io/open-webui/open-webui image, data persisted at /app/backend/data
• Railway private networking — WebUI reaches llama-server via OPENAI_API_BASE_URL=http://llama-server.railway.internal:8080/v1

Deployment Dependencies

• llama.cpp: ggml-org/llama.cpp — GGUF model listings
• Open WebUI: open-webui/open-webui — Docs

🖥️ Minimum Hardware Requirements for llama.cpp + Open WebUI

💡 Start with a 1–4B Q4_K_M model — the sweet spot for quality vs. cost on CPU. Use the server-cuda image with --n-gpu-layers 99 on GPU-enabled services for 10–30× faster inference.

💰 Own Your (Free) Tokens — The Economics

Reality check: you pay for RAM/CPU while running, not tokens. An always-on 8 GB instance is ~$80/mo — heavy API users break even fast. Everyone else can scale down when idle: models reload from the volume in seconds on demand, so light usage costs literal pennies.

🐳 Self-Hosting llama.cpp + Open WebUI Outside Railway

Prefer your own hardware? The same stack runs anywhere Docker runs:

# 1) llama-server with a local GGUF
docker run -d --name llama-server -p 8080:8080 \
  -v ./models:/models \
  ghcr.io/ggml-org/llama.cpp:server \
  llama-server -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 --threads 4 --ctx-size 4096

# 2) Open WebUI, pointed at your server
docker run -d --name open-webui -p 3000:8080 \
  -e OPENAI_API_BASE_URL=http://host.docker.internal:8080/v1 \
  -e OPENAI_API_KEY=local \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000, create your admin account, and chat.

❓ FAQ

How do I change models? Set MODEL_URL to another GGUF URL and redeploy. Each model downloads to the volume once — switching back to a previously downloaded model is instant.

Why llama.cpp instead of Ollama? Raw first-party performance, single-process stability, newest model support on day one, and full control over every inference knob. Open WebUI speaks its OpenAI API natively.

Are my chats and prompts private? Yes — everything stays on your Railway instance and volumes. Nothing is sent to third parties.

Can I run multiple models at once? One model per llama-server process — deploy additional services for more (each with its own volume), and select between them in Open WebUI.

A model feels slow on CPU. Use a smaller Q4_K_M quant, pin --threads to your CPU quota, or use the server-cuda image on a GPU service.

How do I update the engine? Redeploy — the image updates, and your models, chats, and accounts persist on the volumes.