
Deploy Infinity Embeddings + Reranker API
Fast CPU embeddings and reranking API with model cache and API key
Just deployed
Deploy and Host Infinity Embeddings + Reranker API on Railway
Infinity is a high throughput inference server for embedding, reranking and classification models. Version 3 is written in Rust and serves several Hugging Face models from one process, with an OpenAI compatible /v1/embeddings route, a /rerank route, token aware dynamic batching, Prometheus metrics and Swagger docs. This template gives you a private embeddings and reranking backend for RAG and search without paying per token.
About Hosting Infinity Embeddings + Reranker API
Hosting Infinity is a single container with a single volume. This template pins the official CPU image ghcr.io/michaelfeil/infinity_v3:cpu-3.0.1 (234 MB compressed), because the other tags need a GPU and Railway has none. It loads two small models side by side: BAAI/bge-small-en-v1.5 for 384 dimension embeddings and cross-encoder/ms-marco-MiniLM-L6-v2 for reranking. PORT is pinned to 7997 so Railway's healthcheck reaches /health, INFINITY_HOSTNAME is :: so the public domain and other services in your project both reach the server, and the Hugging Face cache sits on a volume mounted at /data, so models download once instead of on every redeploy. A generated INFINITY_API_KEY protects every inference route by default, while /health and /metrics stay public for the platform.
Common Use Cases
- Private embeddings for RAG: index and query with your own endpoint instead of a metered provider
- Two stage retrieval: fetch candidates with embeddings, then reorder them with the reranker on the same service
- Drop in OpenAI embeddings replacement: point any OpenAI SDK at
/v1and change one base URL - Embeddings backend for other Railway services: LiteLLM, Dify, Flowise, n8n, LangChain and LlamaIndex can call
http://infinity.railway.internal:7997/v1
Dependencies for Infinity Embeddings + Reranker API Hosting
- None at runtime: no database, no GPU, no external API keys
- Hugging Face Hub on first boot only, to download about 226 MB of model files into the volume
- Memory: under 1 GB with the default models
- Storage: a volume on
/data, sized for your models
Deployment Dependencies
Implementation Details
The image ENTRYPOINT is the infinity_v3 binary and every setting is an INFINITY_* environment variable, so the template needs no custom start command. Key variables:
INFINITY_MODEL_ID=BAAI/bge-small-en-v1.5;cross-encoder/ms-marco-MiniLM-L6-v2
PORT=7997
INFINITY_PORT=${{PORT}}
INFINITY_HOSTNAME=::
INFINITY_API_KEY=
INFINITY_HUGGINGFACE_HUB_CACHE=/data
INFINITY_MAX_BATCH_TOKENS=8192
First steps after deploy
- Wait for
/healthto return 200. The first boot downloads both models before the port opens. - Copy
INFINITY_API_KEYfrom the service variables. - Embed text. With two models loaded, name the model in every request:
curl -X POST https:///v1/embeddings \
-H "Authorization: Bearer $INFINITY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "BAAI/bge-small-en-v1.5", "input": "Deploying on Railway is easy."}'
- Rerank documents:
curl -X POST https:///rerank \
-H "Authorization: Bearer $INFINITY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "cross-encoder/ms-marco-MiniLM-L6-v2", "query": "persist model weights", "texts": ["A cake recipe.", "Mount a volume for the model cache."]}'
- Call
GET /v1/modelswith the same header to see what is loaded. To change models, editINFINITY_MODEL_ID(semicolon separated) and redeploy; the new weights are cached on the same volume. On the Free plan, serve a single model.
Infinity v3 uses the Infinity Restricted MIT License 1.0, a source available license that allows commercial use and hosting on Railway but not on 25 named providers. Keep the cpu- prefix when you upgrade the image tag.
Why Deploy Infinity Embeddings + Reranker API on Railway?
Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.
By deploying Infinity Embeddings + Reranker API on Railway, you get an authenticated, health checked embeddings and reranking endpoint with managed SSL, a persistent model cache and private networking to the rest of your project, with no per token bill and no GPU to rent.
Template Content

