Lemonade Server

A self-hosted, OpenAI-compatible local-AI server powered by Lemonade (AMD-backed, multi-backend), deployed as a single Railway service. Drop in any HuggingFace GGUF model id and serve chat, completions, embeddings, images, audio, and more via standard OpenAI endpoints.

Deploy and Host

About Hosting

Lemonade Server runs as a single Docker container wrapping the official ghcr.io/lemonade-sdk/lemonade-server:latest image. A thin entrypoint launches the lemond binary on Railway's injected $PORT, exposes a /live healthcheck, and defaults to the CPU backend (Railway's managed containers have no GPU passthrough). Models auto-download from HuggingFace on first request and are cached on a persistent Railway volume, so reboots don't re-fetch weights.

• Default Port: Railway injects $PORT (the server binds it directly)
• Built-in Web UI: Lemonade's own model manager + chat GUI is served at the service root (/) on the same port as the API — open your Railway domain in a browser to use it (no Open WebUI or extra service needed)
• Health Check: GET /live → {"status":"ok"}
• Startup Time: ~10-20 seconds (server boots; model weights download on demand)
• Resource Usage: CPU-only; a 0.6B Q4 model is usable, larger models are slow

Why Deploy

• OpenAI-compatible API — Any OpenAI SDK works against /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, plus image/audio endpoints.
• Zero model baking — No weights in the image; pull any GGUF from HuggingFace at runtime.
• Persistent model cache — A Railway volume at /root/.cache keeps downloaded weights and config across deploys.
• Privacy-first — All inference stays in your Railway project; no third-party API keys required.
• Built-in web UI — Lemonade ships its own browser GUI (model manager + chat) on the same port as the API, so you get a chat interface with no extra services.

Common Use Cases

• Self-hosted chat completion backend for apps expecting OpenAI endpoints
• Built-in chat UI — Open the service URL in a browser for Lemonade's own model manager + chat interface (no separate front-end service needed)
• Private embeddings endpoint for RAG / semantic search
• Local image generation and text-to-speech (TTS) without cloud providers
• A portable OpenAI-compatible API for experimentation and prototyping

Dependencies for

Deployment Dependencies

• HuggingFace reachability — Model weights download from huggingface.co on first load. If egress is blocked, enable Outbound IPv6 in the project's Settings.
• CPU compute — No GPU needed; smaller GGUF models (e.g. Qwen3-0.6B-GGUF) give the best latency on CPU.

API Endpoints

GET /live

Liveness probe for Railway's healthcheck. Returns {"status":"ok"}.

GET /v1/models

List models available locally (those downloaded/cached).

POST /v1/load

Download and load a model by id (auto-fetches from HuggingFace):

{ "model_name": "Qwen3-0.6B-GGUF" }

POST /v1/chat/completions

OpenAI-compatible chat completion. The model auto-downloads on first use if not already loaded.

POST /v1/completions, POST /v1/embeddings

Text completions and embedding vectors, OpenAI-compatible.

Environment Variables

Notes / Limitations

• Railway's managed containers do not expose AMD/NVIDIA devices, so ROCm/Vulkan GPU backends are not available by default — CPU is the reliable path.
• CPU inference is slow and model-size bound. There is no GPU on Railway, so everything runs on CPU. A tiny model like Qwen3-0.6B-GGUF (≈0.4 GB, Q4) gives the best latency (still often 1-3+ minutes per short reply). Models with more parameters or heavier quantization (e.g. 1.5B-3B+) consume more RAM and take exponentially longer per token, and large models can exceed the container's memory and crash or get OOM-killed. Stick to small GGUF models (≤ ~1B params, Q4/Q3) for usable CPU latency.
• The server returns HTTP 200 but generation is CPU-bound — first-token latency and throughput scale directly with model size. Size down if responses feel too slow.
• Railway allows one volume per service; we mount a single volume at /root/.cache covering both the HuggingFace cache and the Lemonade config.

Quick Start

1. Deploy via the button above (or link this repo as a new service).
2. Wait for the build to finish.
3. Load a model (auto-downloads):

   curl -X POST https:///v1/load \
     -H "Content-Type: application/json" \
     -d '{"model_name":"Qwen3-0.6B-GGUF"}'
   

4. Chat with the OpenAI SDK:

   from openai import OpenAI
   client = OpenAI(base_url="https:///v1", api_key="lemonade")
   print(client.chat.completions.create(
       model="Qwen3-0.6B-GGUF",
       messages=[{"role":"user","content":"Hello, Lemonade!"}]
   ).choices[0].message.content)
   

Resources

• Lemonade on GitHub
• Lemonade Server Docs
• Railway Docs