Deploy AI Gateway + Observability (LiteLLM × Langfuse)

Deploy and Host AI Gateway + Observability (LiteLLM × Langfuse) on Railway

One deploy = model gateway + spend tracking + tracing. Route every LLM call in your company through one OpenAI-compatible endpoint — LiteLLM (30k★) proxies 100+ providers (Anthropic, OpenAI, Gemini, Mistral, Bedrock, …) with virtual keys, budgets, and rate limits — and see every request, latency, cost, and full prompt/response in Langfuse (18k★), the leading open-source LLM observability platform. The two are pre-wired: LiteLLM's success and failure callbacks stream straight into your private Langfuse, from the first request.

Swap api.openai.com for your gateway URL and you instantly get: per-team API keys with budgets, provider failover, one bill view across all models, and traces you can actually debug from — without sending a byte of your prompts to a third-party SaaS.

Who it's for: teams running LLM features in production, agencies managing model spend across clients, and anyone tired of guessing what their AI actually costs.

About Hosting AI Gateway + Observability

Seven services, wired over Railway's private network — everything pinned by sha256 digest:

Pre-configured out of the box: Langfuse organization/project/API keys are provisioned headlessly on first boot, and LiteLLM's success_callback/failure_callback reference them over the private network — no clicking through two setup wizards and copy-pasting keys between dashboards. A built-in mock-test model lets you verify the whole gateway→trace pipeline before adding any provider key.

Setup (~5 minutes):

1. Click Deploy Now: set your admin email + password (used for Langfuse login) and optionally paste provider keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY — you can add more later).
2. When services are green, verify the pipeline with the built-in mock model (no provider key needed):

   curl https:///v1/chat/completions \
     -H "Authorization: Bearer " \
     -H "Content-Type: application/json" \
     -d '{"model": "mock-test", "messages": [{"role": "user", "content": "ping"}]}'
   

3. Open the Langfuse URL, log in with your admin credentials — the trace from step 2 is already there, with latency and token counts.
4. Point your apps at the gateway: OpenAI-compatible, so set base_url to your LiteLLM domain and use models like anthropic/claude-opus-4-8 or openai/gpt-4o-mini. Create per-team virtual keys with budgets in the LiteLLM admin UI (/ui, log in with the master key).

Common Use Cases

• One endpoint for every model — switch providers per request (anthropic/…, openai/…, gemini/…) without changing app code
• Team & client budgets — virtual keys with monthly limits and rate caps; see spend per key, per model, per day
• Production LLM debugging — full prompt/response traces with latency and cost, searchable in your own Langfuse
• Provider failover & routing — LiteLLM retries and fallbacks keep features up when one provider degrades
• Agent/workflow observability — point n8n, LangChain, or the Claude Agent SDK at the gateway and every call is traced

Dependencies for AI Gateway + Observability Hosting

• Provider API keys for the models you use (Anthropic, OpenAI, Google, …) — optional at deploy; the mock model works without any
• ~1GB of ClickHouse/MinIO volume storage to start (grows with trace volume; execution data is prunable in Langfuse)

Deployment Dependencies

• Template source (GitHub)
• LiteLLM proxy docs · Langfuse self-host docs
• Anthropic Console · OpenAI Platform

Implementation Details — FAQ & Security

What does it cost to run? Typically $20–35/mo on Railway for the 7 always-on services at moderate trace volume. Compare: gateway SaaS takes ~5% of token spend (OpenRouter) and observability SaaS starts at $59–199/mo (Langfuse Cloud Pro, Helicone, Portkey) with data caps — this stack is flat-cost with your data on your infrastructure.

Why 7 services? Langfuse v3's architecture (worker + ClickHouse + blob storage) is what makes it handle millions of traces; this template mirrors the official self-hosting topology exactly, so upstream docs apply 1:1.

Do traces survive redeploys? Yes — ClickHouse and MinIO run on persistent volumes (single replica each, as required for volumes), and metadata lives in Postgres. LiteLLM and Langfuse web/worker are stateless and safe to redeploy any time.

Can I use it without Langfuse / without LiteLLM? Yes to both: the gateway works standalone (delete the Langfuse services and callbacks), and Langfuse accepts traces directly from any SDK — but the pre-wired pair is the point: gateway-level tracing means zero instrumentation in your apps.

How do upgrades work? Every image is digest-pinned; a weekly automated PR refreshes pins (see PINS.md), reviewed and test-deployed before the template updates. Langfuse v3 minors occasionally ship migrations — they run automatically on boot.

Security notes: master key, salt, encryption keys, and Langfuse secrets are auto-generated per deploy · ClickHouse, MinIO, Redis, and Postgres are private-network only — only LiteLLM and Langfuse web have public URLs · provider keys live in Railway Variables and are additionally encrypted at rest by LiteLLM (salt key) when stored via the UI · treat LITELLM_MASTER_KEY like a root password — create scoped virtual keys for teams.

Why Deploy AI Gateway + Observability on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying AI Gateway + Observability on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

A gateway is only useful if it's always on and close to everything that calls it — on Railway your n8n workflows, agents, and apps reach it over the private network with zero egress, and the whole stack costs less per month than most observability SaaS charges for a single seat.

Built by Bubbles Studio — we build AI automation systems for businesses. Need LLM infrastructure, spend governance, or agents built on top? Get in touch.

More Bubbles templates: WhatsApp AI Receptionist · Claude Agent SDK Worker · n8n Production Stack · Postgres S3 Backup · Webhook Inspector