---
title: "Deploy vLLM | (Just Updated) OpenAI-Compatible LLM API, CPU, Key Login On From Boot"
description: "vLLM OpenAI-compatible API on CPU. API key on from boot, cache on a volume."
category: "AI/ML"
url: https://railway.com/deploy/vllm-or-just-updated-openai-compatible-l
---

# Deploy vLLM | (Just Updated) OpenAI-Compatible LLM API, CPU, Key Login On From Boot

vLLM OpenAI-compatible API on CPU. API key on from boot, cache on a volume.

**[Deploy vLLM | (Just Updated) OpenAI-Compatible LLM API, CPU, Key Login On From Boot on Railway](https://railway.com/template/vllm-or-just-updated-openai-compatible-l)**

Machine-readable deploy manifest (JSON, validated by TemplateCI): https://railway.com/deploy/vllm-or-just-updated-openai-compatible-l/manifest.json

- **Creator:** SuperSlowSloth
- **Category:** AI/ML

## Template content

### vllm

- **Image:** vllm/vllm-openai-cpu:v0.32.0@sha256:ffaba3989b9b96d95a576143c872a4110ce9800f5199b6a36eb27bdff1cbbe05
- **Start command:** `/bin/sh -c 'D=/data; export HF_HOME=$D/huggingface; mkdir -p $HF_HOME
Q=$(cut -d" " -f1 /sys/fs/cgroup/cpu.max 2>/dev/null); P=$(cut -d" " -f2 /sys/fs/cgroup/cpu.max 2>/dev/null)
case "$Q" in ""|max) N=$(nproc);; *) N=$(( (Q+P-1)/P ));; esac
MAX=$(cat /sys/fs/cgroup/memory.max 2>/dev/null)
case "$MAX" in ""|max) MB=8192;; *) MB=$((MAX/1048576));; esac
KVMB=${VLLM_KV_CACHE_MB:-$((MB/16))}; [ "$KVMB" -lt 256 ] && KVMB=256
export OMP_NUM_THREADS=$N MKL_NUM_THREADS=$N VLLM_CPU_OMP_THREADS_BIND=nobind VLLM_ENABLE_V1_MULTIPROCESSING=0
MODEL=${VLLM_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}
PORTN=${PORT:-8080}
echo "[railway] vllm uid=$(id -u) vcpu=$N mem_mb=$MB kv_mb=$KVMB model=$MODEL data_owner=$(stat -c %u:%g $D) shm=$(df -m /dev/shm | tail -1 | tr -s " " | cut -d" " -f2)MB cpu_flags=$(grep -o -w -E "avx512_bf16|avx512f|amx_bf16" /proc/cpuinfo | sort -u | tr "\n" ",")"
BODY=$(printf "{\"model\":\"%s\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hi.\"}],\"max_tokens\":8}" "$MODEL")
( until curl -fs http://127.0.0.1:$PORTN/health >/dev/null 2>&1; do sleep 5; done; echo "[railway] vllm healthy, warming up"; curl -s -m 900 -o /dev/null -w "[railway] warmup chat status=%{http_code} time=%{time_total}s\n" -H "Authorization: Bearer $VLLM_API_KEY" -H "Content-Type: application/json" -d "$BODY" http://127.0.0.1:$PORTN/v1/chat/completions ) &
exec vllm serve "$MODEL" --host 0.0.0.0 --port $PORTN --max-model-len 4096 --kv-cache-memory-bytes $((KVMB*1048576))
'`
- **Public domain:** Yes

## Documentation

# Deploy and Host vLLM on Railway

vLLM is a leading open-source inference server for large language models. This template runs vLLM's CPU build as an OpenAI-compatible API (`/v1/chat/completions`, `/v1/completions`, `/v1/models`) behind an API key that is generated on deploy, with the model cache on a persistent volume.

## About Hosting vLLM

This template runs the official `vllm/vllm-openai-cpu` image (pinned by digest) and serves `Qwen/Qwen2.5-0.5B-Instruct` by default, so a fresh deploy answers chat requests with no GPU and no Hugging Face token. The API key is the generated `VLLM_API_KEY`; vLLM enforces it natively, so every `/v1/*` route rejects an anonymous caller with 401 from the first request, while `/health` stays open for readiness checks. Model weights and the Hugging Face cache live on a volume mounted at `/data`, so a redeploy does not download the model again.

The start script reads the container's real limits from the cgroup instead of the host's: OpenMP and MKL threads are capped at the plan's vCPU count (a CPU inference server sizes its pool from the host's core count otherwise and slows down by a large factor), and the KV cache is fixed at one sixteenth of the memory limit (set `VLLM_KV_CACHE_MB` to change it; the default model fits about 41,000 cached tokens in 476 MB). Railway gives a container a 64 MB `/dev/shm`, which vLLM's default multiprocess engine cannot start in (it asks for 160 MiB and exits with "Insufficient space in /dev/shm"), so the engine runs in-process. The first boot downloads the model and compiles the CPU kernels; allow about three minutes before `/health` returns 200. A background warm-up request is sent once the server is healthy, so the first real request is not the one that pays the compile cost. Measured on Railway's 8 vCPU plan with the default model: about 48 tokens per second after warm-up.

### Agent contract

- Public URL: `https://` followed by the service domain `RAILWAY_PUBLIC_DOMAIN` (variable `VLLM_PUBLIC_URL` holds the full URL)
- Private URL: `http://vllm.railway.internal:8080` (variable `VLLM_PRIVATE_URL`)
- Credentials: variable `VLLM_API_KEY`
- Auth: `Authorization: Bearer $VLLM_API_KEY` on every `/v1/*` call
- Health: `GET /health` returns 200 with no credentials
- Default model: `Qwen/Qwen2.5-0.5B-Instruct`

```bash
# wait for readiness, then list models and run a chat completion
until curl -fs -o /dev/null "$VLLM_PUBLIC_URL/health"; do sleep 10; done
curl -s -o /dev/null -w "anonymous: %{http_code}\n" "$VLLM_PUBLIC_URL/v1/models"
curl -s -H "Authorization: Bearer $VLLM_API_KEY" "$VLLM_PUBLIC_URL/v1/models"
curl -s -H "Authorization: Bearer $VLLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"Say hi."}],"max_tokens":16}' \
  "$VLLM_PUBLIC_URL/v1/chat/completions"
```

To serve a different model, add a service variable `VLLM_MODEL` with a Hugging Face model id and redeploy. Pick a model that fits in the plan's memory; add `HF_TOKEN` for gated models.

## Why Deploy vLLM on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying vLLM on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

- **Locked from the first request** — a generated API key is enforced by vLLM itself; nothing is open to the internet.
- **Model cache survives redeploys** — the volume is preconfigured, so weights download once.
- **Zero configuration** — no variables to fill in and no Hugging Face token needed for the default model.
- **Drop-in OpenAI API** — point any OpenAI SDK at the public URL and the key.

## Common Use Cases

- **Private LLM endpoint for agents** — a stable OpenAI-compatible base URL that coding agents and workflows call.
- **Prototyping and CI for LLM apps** — a cheap, key-protected model server to test prompts and integrations against.
- **Self-hosted inference behind your own services** — call it over the private network from other Railway services.

## Dependencies for vLLM Hosting

- A volume mounted at `/data` (included in this template).

### Deployment Dependencies

- [vLLM](https://github.com/vllm-project/vllm) — upstream server
- [vllm/vllm-openai-cpu on Docker Hub](https://hub.docker.com/r/vllm/vllm-openai-cpu) — the pinned image this template deploys
- [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) — default model (Apache-2.0)


## Similar templates

- [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform.
- [Möbius v0.1 Personal AI OS](https://railway.com/deploy/mobius-v01-personal-ai-os) — Self-hosted AI agent that builds and runs apps for you, with memory.
- [stella](https://railway.com/deploy/stella) — Self-host stella with web, API, Postgres, Redis, and object storage.

Open this page in a browser: https://railway.com/deploy/vllm-or-just-updated-openai-compatible-l