--- title: "Deploy LiteLLM Proxy — Self-Hosted AI Gateway for 100+ LLMs" description: "Self-host an AI gateway for 100+ LLMs with spend tracking & budgets." category: "AI/ML" url: https://railway.com/deploy/litellm-ai-gateway --- # Deploy LiteLLM Proxy — Self-Hosted AI Gateway for 100+ LLMs Self-host an AI gateway for 100+ LLMs with spend tracking & budgets. **[Deploy LiteLLM Proxy — Self-Hosted AI Gateway for 100+ LLMs on Railway](https://railway.com/template/litellm-ai-gateway)** - **Creator:** SilverBanana - **Category:** AI/ML - **Total deploys:** 2 ## Template content ### LiteLLM https://bogusz.co/external/litellm.png - **Image:** berriai/litellm:main-stable ### Redis https://cdn.sanity.io/images/sy1jschh/production/0ce0bfdcfbdbf69662b1116671f97c2dd788b655-157x157.svg - **Image:** redis:8.2.1 - **Start command:** `/bin/sh -c "rm -rf $RAILWAY_VOLUME_MOUNT_PATH/lost+found/ && exec docker-entrypoint.sh redis-server --requirepass $REDIS_PASSWORD --save 60 1 --dir $RAILWAY_VOLUME_MOUNT_PATH"` ### Postgres https://devicons.railway.app/i/postgresql.svg - **Image:** ghcr.io/railwayapp-templates/postgres-ssl:18 ## Documentation # Deploy and Host LiteLLM Proxy on Railway LiteLLM Proxy is the open-source AI gateway — **~40k GitHub stars**, YC-backed, 240M+ Docker pulls — that puts one OpenAI-compatible endpoint in front of 100+ LLM providers. Point your apps at a single URL and route to OpenAI, Anthropic, Gemini, Azure, Groq, Bedrock, or a local Ollama behind the scenes — with virtual keys, per-team and per-user spend tracking, budget enforcement, load balancing, fallbacks, and an admin dashboard out of the box. This template deploys LiteLLM with PostgreSQL (for virtual keys and spend tracking) and Redis (for caching) — the production setup. Self-host on Railway for ~$10–15/month and replace Helicone ($20/mo), OpenRouter's markup, and hand-rolled key-management scripts with infrastructure you own. --- ## What This Template Deploys | Service | Purpose | |---|---| | **LiteLLM Proxy** | The AI gateway — unified OpenAI-compatible API for 100+ providers, virtual keys, spend tracking, and admin UI on port `4000` | | **PostgreSQL 16** | Stores virtual keys, per-key/team spend, budgets, and request metadata — required for the dashboard and key management | | **Redis** | Caches responses and coordinates rate limiting and routing across requests | All services connect over Railway's private network with credentials injected automatically. The proxy is secure by default — a master key gates all requests. Point any OpenAI SDK at your Railway URL with a one-line `base_url` change and existing code works unchanged. --- ## About Hosting LiteLLM Proxy The problem LiteLLM solves: several developers, multiple providers, and no visibility into who's burning budget on which model until the invoice lands. LiteLLM exposes one OpenAI-compatible endpoint, routes each request to the provider you configured, tracks spend per virtual key, enforces budgets *before* the request leaves your server, and logs everything — in one service. It needs PostgreSQL (virtual keys, spend tracking) and ideally Redis (caching). LiteLLM is CPU-only Python at ~400 MB idle — no GPU required, which makes Railway a natural home. This template wires the proxy, Postgres, and Redis over private networking with automatic HTTPS. Typical cost: **~$10–15/month** on Railway for the proxy, PostgreSQL, and Redis, plus your own LLM provider usage. That replaces Helicone's $20/month observability, OpenRouter's per-request markup, and the hand-rolled key scripts teams otherwise maintain — while keeping all keys and spend data on your own infrastructure. --- ## Deploy in Under 5 Minutes 1. Click **Deploy on Railway** — LiteLLM, PostgreSQL, and Redis build automatically (~3–4 minutes) 2. `DATABASE_URL`, `REDIS_URL`, and a `LITELLM_MASTER_KEY` (starting with `sk-`) are wired/generated 3. Add provider API keys as variables — `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, etc. 4. Open `[your-domain]/ui` and log in with `admin` and your master key to reach the dashboard 5. Point your app's OpenAI client `base_url` at your Railway URL and start routing requests No per-provider SDKs. No key-management scripts. No code changes beyond the base URL. --- ## Common Use Cases - **One API for 100+ LLM providers** — call OpenAI, Anthropic, Gemini, Groq, Bedrock, and local Ollama through a single OpenAI-compatible endpoint; switch models with a config change, no code - **Per-team and per-user spend tracking** — issue virtual keys with budgets so you see exactly which project burns which model's budget, and cap spend before the invoice surprises you - **Self-hosted alternative to Helicone / OpenRouter** — get observability and multi-provider routing without Helicone's $20/month or OpenRouter's per-request markup, on your own instance - **Automatic fallbacks and load balancing** — route a primary model with a fallback list so a provider 429 or outage transparently retries on the next provider with no app changes - **Front your Ollama and cloud models together** — expose local Ollama models and cloud APIs behind one gateway, so apps use the same interface for self-hosted and hosted inference - **Budget guardrails for AI features** — enforce hard spend limits per key; LiteLLM returns a 429 before forwarding when a budget is exceeded, so no runaway costs reach the provider --- ## Configuration | Variable | Required | Description | |---|---|---| | `LITELLM_MASTER_KEY` | ✅ Required | Admin/master key — must start with `sk-`; gates all requests and creates virtual keys | | `DATABASE_URL` | ✅ Auto-injected | PostgreSQL connection string via Railway reference variable | | `REDIS_URL` | Recommended | Redis URI for caching and rate limiting via Railway reference variable | | `OPENAI_API_KEY` | Optional | Enables OpenAI models (add keys only for providers you use) | | `ANTHROPIC_API_KEY` | Optional | Enables Anthropic Claude models | | `GEMINI_API_KEY` | Optional | Enables Google Gemini models | | `STORE_MODEL_IN_DB` | Recommended | `true` — lets you add/manage models from the dashboard, stored in Postgres | | `PORT` | Pre-set | `4000` — the proxy's default port | > The master key must start with `sk-` and gates everything — keep it secret and use virtual keys > (generated from it) for apps and teams, never the master key directly. Add provider keys only for > the providers you actually use; each one auto-registers that provider's models. --- ## LiteLLM Proxy vs. Alternatives | | LiteLLM (Railway) | OpenRouter | Helicone | Direct per-provider | |---|---|---|---|---| | **Cost** | ~$10–15/mo flat | Per-request markup | $20/mo+ | Provider fees only | | **Self-hosted / own keys** | ✅ Yes | ❌ Their infra | ⚠️ Proxy/SaaS | ✅ Yes | | **100+ providers, one API** | ✅ Yes | ✅ Yes | ⚠️ Observability | ❌ Separate SDKs | | **Virtual keys + budgets** | ✅ Built-in | ⚠️ Limited | ⚠️ Tracking only | ❌ Build it | | **Spend tracking per team** | ✅ Yes | ⚠️ Basic | ✅ Yes | ❌ No | | **Fallbacks + load balancing** | ✅ Yes | ✅ Yes | ❌ No | ❌ Build it | | **Data ownership** | ✅ Your instance | ❌ OpenRouter | ⚠️ Varies | ✅ Yes | | **Open source** | ✅ MIT | ❌ No | ⚠️ Partial | N/A | --- ## Dependencies for LiteLLM Hosting - Railway account — Hobby plan (~$10–15/month) for the proxy, PostgreSQL, and Redis - API keys for the LLM providers you want to route to (added as variables) - Any OpenAI-compatible client — no LiteLLM-specific SDK required ### Deployment Dependencies - [LiteLLM GitHub Repository](https://github.com/BerriAI/litellm) — source and releases - [LiteLLM Documentation](https://docs.litellm.ai) — proxy config, virtual keys, and routing - [LiteLLM Virtual Keys Docs](https://docs.litellm.ai/docs/proxy/virtual_keys) — key and budget management ### Implementation Details This template deploys the LiteLLM proxy with PostgreSQL 16 and Redis over Railway's private network. LiteLLM requires PostgreSQL for the admin UI, virtual keys, and spend tracking — an in-memory mode exists but drops those core features, so Postgres is used here; Redis handles caching. The proxy is secure by default: `LITELLM_MASTER_KEY` (must start with `sk-`) authenticates all requests, and you generate scoped virtual keys from it for apps and teams. Configure models via `config.yaml` or, with `STORE_MODEL_IN_DB=true`, from the dashboard. Point any OpenAI client's `base_url` at your Railway URL — the API shape is identical to OpenAI's, so migration is one line. --- ## Frequently Asked Questions **What does LiteLLM Proxy actually do?** It's a self-hosted AI gateway: one OpenAI-compatible endpoint in front of 100+ LLM providers. Your apps talk to LiteLLM, and it routes to whichever provider/model you configured, tracks spend per virtual key, enforces budgets, and handles fallbacks — so you stop wiring provider-specific SDKs and get central visibility and control over LLM usage. **How does it save money versus OpenRouter or Helicone?** OpenRouter adds a per-request markup; Helicone's observability starts at $20/month. LiteLLM self-hosted is ~$10–15/month flat on Railway and you pay providers directly at their rates with no markup. You also prevent overspend: budgets return a 429 before a request reaches the provider, so no cost is incurred once a key is over budget. **Does it need PostgreSQL?** Yes for the full feature set. Virtual keys, per-team spend tracking, and the admin dashboard all require PostgreSQL. LiteLLM can run in-memory without a database, but that drops the spend and key features that are the main reason to run the proxy — so this template includes Postgres. **Can I use it with local Ollama and cloud models together?** Yes. Configure Ollama models and cloud providers in the same LiteLLM instance, and apps access both through the identical OpenAI-compatible interface. It pairs naturally with a self-hosted Ollama on Railway — LiteLLM becomes the unified front door for local and cloud inference. **Do I lose keys and spend data if Railway redeploys?** No. Virtual keys, budgets, spend history, and model config (with `STORE_MODEL_IN_DB=true`) live in the Railway-managed PostgreSQL database. Redeploys and updates don't affect them. **Is it hard to migrate my app to LiteLLM?** No — it's a one-line change. LiteLLM's API matches OpenAI's, so you point your client's `base_url` at your Railway proxy URL and use a LiteLLM virtual key. Everything else stays the same. --- ## Why Deploy and Host LiteLLM Proxy on Railway? Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it. By deploying LiteLLM Proxy on Railway, you get a self-hosted AI gateway for 100+ LLM providers — one OpenAI-compatible API, virtual keys, per-team spend tracking, budgets, and fallbacks — backed by PostgreSQL and Redis at ~$10–15/month flat, with all your keys and usage data on infrastructure you own. ## Similar templates - [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform. - [stella](https://railway.com/deploy/stella) — Self-host stella with web, API, Postgres, Redis, and object storage. - [Hermes Agent | OpenClaw Alternative with Dashboard](https://railway.com/deploy/hermes-agent-or-openclaw-alternative-wit) — Self-Hosted Hermes AI Agent for Telegram, Discord & Slack Open this page in a browser: https://railway.com/deploy/litellm-ai-gateway