Deploy Text Embeddings Inference + Model Cache

Deploy and Host Text Embeddings Inference on Railway

Text Embeddings Inference (TEI) is Hugging Face's Rust inference server for embedding, reranking and sequence classification models. It turns any of the thousands of models tagged text-embeddings-inference on the Hugging Face Hub into a fast HTTP API with an OpenAI compatible /v1/embeddings route, token aware dynamic batching, Prometheus metrics and Swagger docs, so you can drop it in wherever you would otherwise pay per token for embeddings.

About Hosting Text Embeddings Inference

Hosting TEI is a single container with a single volume. This template pins the official CPU image ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.4, because the default tags are CUDA builds and Railway has no GPUs. PORT is pinned to 3000 so Railway's healthcheck reaches /health, HOSTNAME is :: so the public domain and other services in your project both reach the server over IPv6 private networking, and HUGGINGFACE_HUB_CACHE points at a volume mounted on /data so the model is downloaded once instead of on every redeploy. The healthcheck window is 600 seconds because the port only opens after the weights are downloaded and loaded. A generated API_KEY protects every inference route by default, while /health and /metrics stay public so the platform can probe the service.

Common Use Cases

• Private embeddings for RAG: index and query with your own endpoint instead of a metered provider
• Drop-in OpenAI embeddings replacement: point any OpenAI SDK at /v1 and change one base URL
• Reranking search results: set MODEL_ID to a cross encoder such as BAAI/bge-reranker-base and call /rerank
• Zero shot classification and scoring: sequence classification models serve /predict
• Embeddings backend for other Railway services: LiteLLM, LibreChat, Dify, Flowise and n8n all accept an OpenAI compatible embeddings URL at http://text-embeddings-inference.railway.internal:3000/v1

Dependencies for Text Embeddings Inference Hosting

• None at runtime: no database, no queue, no GPU, no external API keys
• Hugging Face Hub on first boot only, to download the model into the volume
• Memory: about 1 to 2 GB with the default model (Hobby plan or higher)
• Storage: a volume on /data, sized for your model (135 MB for the default)

Deployment Dependencies

• Text Embeddings Inference repository
• Official documentation
• Supported models
• BAAI/bge-small-en-v1.5 model card

Implementation Details

The image ENTRYPOINT is the text-embeddings-router binary and every setting has an environment variable, so the template needs no custom start command. Key variables:

MODEL_ID=BAAI/bge-small-en-v1.5
PORT=3000
HOSTNAME=::
API_KEY=
HUGGINGFACE_HUB_CACHE=/data
MAX_BATCH_TOKENS=16384
MAX_CLIENT_BATCH_SIZE=32

First steps after deploy

1. Wait for /health to return 200. First boot downloads about 135 MB before the port opens.
2. Copy API_KEY from the service variables.
3. Call the API:

curl -X POST https:///v1/embeddings \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input": "Deploying on Railway is easy.", "model": "tei"}'

4. Check GET /info with the same header to see the model, dtype and maximum input length.
5. To change models, set MODEL_ID to another Hub id and redeploy. The new weights are cached on the same volume.

Keep the cpu- prefix when you upgrade the image tag. latest and the GPU tags cannot start on Railway. If the container runs out of memory, lower MAX_BATCH_TOKENS before moving to a bigger plan.

Why Deploy Text Embeddings Inference on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Text Embeddings Inference on Railway, you get an authenticated, health checked, OpenAI compatible embeddings endpoint with managed SSL, a persistent model cache and private networking to the rest of your project, with no per token bill and no GPU to rent.