Deploy an LLM Gateway with LiteLLM
Deploy LiteLLM Proxy on Railway as an LLM gateway for multiple providers. Covers model routing, cost tracking, API key management, caching with Redis, and private networking.
Last updated: November 2026
As AI applications scale, managing multiple LLM providers becomes complex. Different services may use different providers, API keys rotate, costs are hard to track, and a single provider outage can take down your entire application.
An LLM gateway (sometimes called an AI API gateway) sits between your services and LLM providers. It provides a single endpoint with a unified API, model routing, cost tracking, rate limiting, and provider failover.
LiteLLM Proxy is the most widely used open-source option. It exposes an OpenAI-compatible API that routes requests to 100+ LLM providers.
Architecture
- LiteLLM Proxy runs as a Railway service with no public domain. Other services in the project call it over private networking.
- Redis (optional) provides response caching to reduce API costs and latency.
- Postgres (optional) stores request logs and cost data for analytics.
Your application services send requests to the proxy's internal URL instead of directly to OpenAI, Anthropic, or other providers.
Prerequisites
- A Railway account
- API keys for one or more LLM providers
1. Create the proxy repository
Create a new repository with two files:
litellm_config.yaml:
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
- model_name: claude-opus
litellm_params:
model: anthropic/claude-opus-4-8
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_key: os.environ/OPENAI_API_KEYDockerfile:
FROM ghcr.io/berriai/litellm:main-latest
COPY litellm_config.yaml /app/config.yaml
CMD ["--config", "/app/config.yaml", "--port", "8000", "--host", "0.0.0.0"]2. Deploy the proxy
To skip the manual setup, deploy the LiteLLM template, which comes set up for virtual keys, budgets, and cost tracking. To build it yourself:
- Create a new project on Railway.
- Click + New > GitHub Repo and select your proxy repository.
- Set the following environment variables on the proxy service:
| Variable | Value |
|---|---|
OPENAI_API_KEY | Your OpenAI API key |
ANTHROPIC_API_KEY | Your Anthropic API key |
LITELLM_MASTER_KEY | A secret key for proxy admin access. It must start with sk-. |
- Railway builds the Dockerfile and starts the proxy.
3. Keep the proxy internal
The proxy should not be publicly accessible. Do not generate a public domain for it. Other services in the same project reach it via private networking at:
http://litellm-proxy.railway.internal:8000Replace litellm-proxy with your service name. The port is 8000 because the Dockerfile starts LiteLLM with --port 8000. If you change the port there, change it here too.
4. Connect your services
Update your application services to point at the proxy instead of directly at LLM providers. Since LiteLLM exposes an OpenAI-compatible API, you only need to change the base URL:
from openai import OpenAI
client = OpenAI(
base_url="http://litellm-proxy.railway.internal:8000/v1",
api_key="your-litellm-master-key",
)
response = client.chat.completions.create(
model="gpt-4o", # Routed by the proxy to the configured provider
messages=[{"role": "user", "content": "Hello"}],
)Any OpenAI SDK client (Python, Node.js, Go) works with the proxy by changing base_url.
5. Add Redis for caching
Response caching reduces costs by returning cached results for identical requests:
- Add Redis to your project.
- Add the Redis connection to the proxy's environment variables:
| Variable | Value |
|---|---|
REDIS_HOST | ${{Redis.REDISHOST}} |
REDIS_PORT | ${{Redis.REDISPORT}} |
REDIS_PASSWORD | ${{Redis.REDISPASSWORD}} |
- Enable caching in your
litellm_config.yaml:
litellm_settings:
cache: true
cache_params:
type: redis6. Track costs
LiteLLM logs request metadata including token counts and estimated costs. To persist this data, add Postgres and set:
| Variable | Value |
|---|---|
DATABASE_URL | ${{Postgres.DATABASE_URL}} |
LiteLLM automatically creates its logging tables on first connection. Access cost data through the LiteLLM admin UI or query the database directly.
7. Issue virtual keys with spend limits
Instead of giving every service the master key, give each service or team its own virtual key with a budget. Virtual keys need the Postgres database from the previous step (DATABASE_URL) and a LITELLM_MASTER_KEY that starts with sk-.
The proxy has no public domain, so generate keys from another service in the project, using the master key:
curl http://litellm-proxy.railway.internal:8000/key/generate \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"models": ["gpt-4o-mini", "claude-opus"], "max_budget": 25, "budget_duration": "30d"}'max_budgetcaps the key's spend, in US dollars.budget_durationresets that spend on a schedule, here every 30 days.modelslimits which model names fromlitellm_config.yamlthe key can call.
The response contains a new key that starts with sk-. Give it to the calling service as its api_key in place of the master key, and keep the master key on the proxy only. Check a key's spend with /key/info and raise its budget with /key/update. See LiteLLM's virtual keys docs for rate limits and team budgets.
Next steps
- Build an LLM Gateway in TypeScript: the build-your-own route, when the logic you need is small and specific.
- Deploy an AI-Powered SaaS App: Build a product that uses the gateway.
- Private Networking: How services communicate within a project.
- Redis on Railway: Persistence settings and memory management.
- PostgreSQL on Railway: Connection pooling, backups, and configuration.