Deploy Xinference — Self-Hosted Embeddings, Rerankers & LLMs

Deploy and Host Xinference on Railway

Xinference (Xorbits Inference) is an open-source model-serving platform that puts LLMs, embedding models, and rerankers behind one OpenAI-compatible API — swap OpenAI for a self-hosted endpoint without rewriting your app. This template deploys the CPU build with a persistent volume for model weights and built-in authentication, sized for what runs well on Railway: fast embedding and reranker models for RAG pipelines, plus small quantized LLMs. Point any OpenAI SDK at your own private inference endpoint, on infrastructure you own.

What This Template Deploys

A single service — no external database. A persistent volume at /data holds the model registry, virtual environments, and HuggingFace cache, so downloaded models survive redeploys. Built-in JWT and API-key auth secure the endpoint from boot.

About Hosting

Xinference is a powerful model server, and the key to a good Railway deployment is running the right kind of model — this template is set up around that.

Best on Railway for embeddings and rerankers — the RAG workhorses. Railway is CPU-only (no GPU), and embedding and reranker models are small, fast, and run well on CPU. Models like bge-small-en-v1.5 (embeddings) and bge-reranker-base (reranking) give you a private, self-hosted endpoint for the retrieval side of any RAG pipeline — no per-call embedding fees, your data staying on your infrastructure. This is the strongest fit for the platform, and the reason to run Xinference here.

Small quantized LLMs run too — right-size the plan. You can serve small Q4-quantized LLMs (e.g. a 7B Q4_K_M model), but CPU inference is slower than GPU and RAM-bound: anything beyond a ~7B Q4 model needs more than 8 GB RAM and will OOM at launch. Pick smaller models or smaller quants, and size your Railway plan to the largest model you intend to run. For heavy, low-latency LLM serving, a GPU host is the right home; for a private endpoint and light generation, Railway works.

One OpenAI-compatible endpoint for many models. Once a model is running, point any OpenAI SDK at https://your-domain/v1 with your API key as the bearer token — chat, completions, and embeddings all work through the standard interface, so you swap providers without changing code. It integrates natively with LangChain, LlamaIndex, and Dify.

Authenticated from boot. The endpoint is protected with JWT and an XINFERENCE_API_KEY rendered from your environment variables, and the web UI is guarded by XINFERENCE_ADMIN_USER and XINFERENCE_ADMIN_PASSWORD. Set strong values — this is a public endpoint that serves models.

Models persist on the volume. Downloaded weights, the model registry, and the HuggingFace cache live on the /data volume, so they survive redeploys instead of re-downloading each time. Manage models from the web UI at your domain with /ui/ appended (note the trailing slash).

Typical cost: ~$5–15/month on Railway depending on the model size and RAM you provision. Xinference is Apache-2.0 licensed and free.

How It Compares

The OpenAI API is simplest but bills per token and holds your data. A GPU inference host is right for heavy, low-latency LLM serving but costs more. Xinference on Railway's sweet spot is a private, OpenAI-compatible endpoint for the CPU-friendly models — embeddings and rerankers for RAG, plus light LLM use — self-hosted at flat cost with your data on infrastructure you own. Pair it with a vector database for a fully self-hosted retrieval stack.

Deploy in Under 5 Minutes

1. Click Deploy on Railway — the CPU Xinference service builds with a volume for models (~3 minutes)
2. Set XINFERENCE_API_KEY, XINFERENCE_ADMIN_USER, and XINFERENCE_ADMIN_PASSWORD
3. Open your Railway domain with /ui/ appended and log in
4. Launch a model — start with bge-small-en-v1.5 (embeddings) or bge-reranker-base (reranking)
5. Point your OpenAI SDK at https://your-domain/v1 with your API key and start calling it

Right-size your plan's RAM to the largest model you launch to avoid out-of-memory errors.

Common Use Cases

• Self-hosted embeddings for RAG — a private embedding endpoint for your retrieval pipeline, no per-call fees
• Reranking — improve RAG relevance with a self-hosted reranker like bge-reranker-base
• Light LLM serving — run small quantized models for low-volume generation on your own infra
• Data-sovereign inference — keep embedding and inference requests on infrastructure you control

Configuration

Right-size RAM to your model. Railway is CPU-only, so favor embeddings and rerankers, and use only small Q4 LLMs — anything beyond ~7B Q4 needs more than 8 GB RAM and will OOM. Size the plan to the largest model you launch.

Set strong auth. XINFERENCE_API_KEY secures the API and the admin credentials guard the UI — this is a public endpoint serving models, so use strong values.

Dependencies for Xinference Hosting

• Railway account — ~$5–15/month depending on model size and RAM
• A persistent volume at /data for model weights and cache (included)
• Enough RAM for your chosen model (embeddings/rerankers are light; LLMs need more)
• An OpenAI-compatible client, LangChain, LlamaIndex, or Dify to call the endpoint

Deployment Dependencies

• Xinference GitHub Repository
• Xinference Documentation
• Supported Models
• Railway Volumes Documentation

Implementation Details

The template runs the official xprobe/xinference:latest-cpu image as a single service on port 9997, with a persistent volume mounted at /data (XINFERENCE_HOME) holding the model registry, per-model virtual environments, and the HuggingFace cache, so downloaded weights survive redeploys. There is no external database. Authentication is enabled from boot: a JWT and XINFERENCE_API_KEY protect the API, and XINFERENCE_ADMIN_USER / XINFERENCE_ADMIN_PASSWORD guard the web UI, reachable at the Railway domain with /ui/ appended.

Because Railway provides CPU compute rather than GPUs, the template is oriented toward models that perform well on CPU — embedding models (e.g. bge-small-en-v1.5) and rerankers (e.g. bge-reranker-base) for retrieval pipelines — with small Q4-quantized LLMs also runnable but RAM-bound; models beyond roughly 7B Q4_K_M exceed 8 GB and fail to launch, so the plan should be sized to the largest intended model. Xinference wraps llama.cpp, transformers, and sentence-transformers behind one OpenAI-compatible API, so any OpenAI SDK can call /v1 with the bearer token, and it integrates with LangChain, LlamaIndex, and Dify. Model data on the volume is the component to back up.

Frequently Asked Questions

Can I run large LLMs on this? Not heavy ones — Railway is CPU-only, so large models are slow and RAM-bound, and anything beyond a ~7B Q4 model exceeds 8 GB and won't launch. Railway is best here for embeddings, rerankers, and small quantized LLMs; use a GPU host for heavy LLM serving.

What's the best fit for Railway? Embedding and reranker models for RAG — they're small, fast on CPU, and give you a private, self-hosted endpoint for the retrieval side of your pipeline without per-call fees.

Is the endpoint secured? Yes — a JWT and XINFERENCE_API_KEY protect the API, and admin credentials guard the web UI, all from boot. Set strong values since it's a public endpoint.

How do I call it from my app? Point any OpenAI SDK at https://your-domain/v1 with your API key as the bearer token. Chat, completions, and embeddings work through the standard OpenAI interface, and it integrates with LangChain, LlamaIndex, and Dify.

Do my models persist? Yes — weights, the registry, and the cache live on the /data volume, so they survive redeploys instead of re-downloading.

How do I avoid out-of-memory errors? Size your Railway plan's RAM to the largest model you launch. Embeddings and rerankers are light; for LLMs, use small quantized models and provision enough RAM.

Why Deploy Xinference on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Xinference on Railway you get a private, OpenAI-compatible inference endpoint with the setup handled — the CPU image, a persistent volume for models, and authentication from boot. Ideal for self-hosted embeddings and rerankers in your RAG stack, plus light LLM serving, on infrastructure you own.