---
title: "Deploy llama.cpp Server"
description: "llama.cpp server with an OpenAI-compatible API running GGUF models on CPU."
category: "AI/ML"
url: https://railway.com/deploy/llamacpp-server
---

# Deploy llama.cpp Server

llama.cpp server with an OpenAI-compatible API running GGUF models on CPU.

**[Deploy llama.cpp Server on Railway](https://railway.com/template/llamacpp-server)**

Machine-readable deploy manifest (JSON, validated by TemplateCI): https://railway.com/deploy/llamacpp-server/manifest.json

- **Creator:** Agaz Self-Host
- **Category:** AI/ML

## Template content

### llamacpp https://github.com/ggml-org.png

- **Image:** ghcr.io/ggml-org/llama.cpp:server-v0.5.0
- **Health check:** /health
- **Public domain:** Yes

## Documentation

# Deploy and Host llama.cpp Server on Railway

llama.cpp is the widely used C/C++ engine for running large language models locally, and `llama-server` is its lightweight HTTP server. It loads GGUF models, offers an OpenAI-compatible chat and completions API with streaming, tool calling and JSON schema output, and runs efficiently on CPUs without a GPU.

## About Hosting llama.cpp Server

This template deploys the official llama.cpp server image (v0.5.0) on CPU with `ggml-org/Qwen3-0.6B-GGUF` as the default model. The model is downloaded from Hugging Face on first start into a Railway volume and reused on restarts. Every request needs the generated API key. The thread count is pinned to 4: containers see the host's cores, and the default over-subscribed the CPU quota and slowed generation to a crawl. With 4 threads the small model generated about 70 to 90 tokens per second in testing. Larger models need more memory and a bigger plan.

## Common Use Cases

- A private OpenAI-compatible endpoint for small open models
- Offline or low-cost LLM features such as classification, extraction and summaries
- Testing prompts and tools against local models before scaling up

## Dependencies for llama.cpp Server Hosting

- `ghcr.io/ggml-org/llama.cpp:server-v0.5.0` (official image)
- A Railway volume at `/models` for downloaded GGUF files

### Deployment Dependencies

- [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/tree/master/tools/server)
- [llama.cpp v0.5.0 release](https://github.com/ggml-org/llama.cpp/releases/tag/v0.5.0)
- [GGUF models on Hugging Face](https://huggingface.co/models?library=gguf)
- [Railway volumes](https://docs.railway.com/reference/volumes)

### Implementation Details

| Service | Image | Networking | Storage |
| --- | --- | --- | --- |
| llamacpp | `llama.cpp:server-v0.5.0` | public domain on 8080; private IPv4/IPv6 | volume at `/models` |

```bash
curl "$OPENAI_BASE_URL/chat/completions" -H "Authorization: Bearer $LLAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-0.6b", "messages": [{"role": "user", "content": "Hello"}]}'
```

Any OpenAI SDK works with `base_url=${{llamacpp.OPENAI_BASE_URL}}` and `api_key=${{llamacpp.LLAMA_API_KEY}}`.

| Variable | Default | Purpose |
| --- | --- | --- |
| `LLAMA_ARG_HF_REPO` | `ggml-org/Qwen3-0.6B-GGUF` | Model repo (append `:Q4_K_M` to pick a quant) |
| `LLAMA_ARG_ALIAS` | `qwen3-0.6b` | Model name reported by the API |
| `LLAMA_ARG_THREADS` | `4` | CPU threads; keep at or below your vCPUs |
| `LLAMA_ARG_CTX_SIZE` | `4096` | Context window |
| `LLAMA_API_KEY` | generated | Required bearer token |

Notes:

- Every `llama-server` flag has a `LLAMA_ARG_*` variable; see the server documentation.
- Add `/no_think` to Qwen3 prompts to skip the reasoning block for short answers.

This is a community-maintained deployment package and does not imply affiliation with or endorsement by the llama.cpp Server project or its maintainers.

## Why Deploy llama.cpp Server on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying llama.cpp Server on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.


## Similar templates

- [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform.
- [stella](https://railway.com/deploy/stella) — Self-host stella with web, API, Postgres, Redis, and object storage.
- [Hermes Agent | OpenClaw Alternative with Dashboard](https://railway.com/deploy/hermes-agent-or-openclaw-alternative-wit) — Self-Hosted Hermes AI Agent for Telegram, Discord & Slack

Open this page in a browser: https://railway.com/deploy/llamacpp-server