Deploy Squirrel

Unified LLM API gateway with smart routing and failover.

Deploy Squirrel

Just deployed

/data

Deploy and Host Squirrel LLM Gateway on Railway

Squirrel LLM Gateway is an open-source, self-hosted AI API gateway that unifies access to multiple Large Language Model (LLM) providers. It offers intelligent model routing, automatic failover, load balancing, API key management, request logging, and cost analytics. Squirrel supports OpenAI-compatible and Anthropic-compatible APIs through a centralized management dashboard.

About Hosting Squirrel LLM Gateway

Hosting Squirrel LLM Gateway on Railway provides a centralized API endpoint for connecting applications to multiple AI providers without managing complex infrastructure.

This template uses the official Docker image with a lightweight, single-service architecture powered by SQLite. Railway handles container deployment, HTTPS networking, and persistent storage through an attached volume.

The gateway runs on port 8000 and includes a web-based administration dashboard for managing providers, model mappings, API keys, routing policies, and usage analytics.

Once deployed, applications can connect to Squirrel using OpenAI-compatible SDKs and API endpoints, while the gateway handles upstream provider communication, retries, and request routing.

Common Use Cases

  • Unified AI API Gateway: Connect multiple LLM providers through a single API endpoint without maintaining separate integrations.
  • Intelligent Model Routing: Route requests based on model names, priorities, weights, costs, and configurable routing rules.
  • Automatic Provider Failover: Improve application reliability through automatic retries and fallback providers.
  • API Key Management: Issue gateway API keys while keeping upstream provider credentials centralized.
  • AI Usage Monitoring: Track requests, token consumption, latency, and estimated API costs through a management dashboard.
  • Multi-Provider AI Applications: Provide a consistent API interface for AI agents, chatbots, automation workflows, and backend applications.

Dependencies for Squirrel LLM Gateway Hosting

  • Squirrel Docker Image: Official container image ghcr.io/mylxsw/llm-gateway, exposing HTTP port 8000.
  • SQLite Database: Built-in database for storing providers, model mappings, API keys, and application configuration.
  • Persistent Storage: One Railway volume mounted at /data to retain SQLite data across restarts and redeployments.
  • LLM Provider: At least one external or self-hosted AI provider with an accessible API endpoint and valid credentials.
  • Railway Service: A single container service with public HTTP networking enabled.
  • Redis: Not required for this template.

Squirrel LLM Gateway vs. Alternatives

Squirrel focuses on unified LLM access, intelligent routing, and observability with a relatively lightweight deployment architecture.

FeatureSquirrel LLM GatewayLiteLLM ProxyNew API
Primary FocusIntelligent LLM routingMulti-provider AI gatewayAI API management
Self-Hosted✅✅✅
OpenAI-Compatible API✅✅✅
Multiple LLM Providers✅✅✅
Web Management Dashboard✅✅✅
API Key Management✅✅✅
Load Balancing✅✅✅
Automatic Failover✅✅⚠️ Configuration-dependent
Usage & Cost Analytics✅✅✅
SQLite Support✅✅✅
Single-Container Deployment✅✅✅
Advanced Routing Rules✅✅⚠️
Deployment ComplexityLow–MediumMediumLow–Medium

Choose Squirrel LLM Gateway for intelligent routing, failover, and detailed AI request observability through a unified gateway.

Choose LiteLLM Proxy for extensive model provider compatibility, AI infrastructure integrations, and advanced gateway functionality.

Choose New API for centralized API distribution, provider channel management, token administration, and usage tracking.

How to Use Squirrel LLM Gateway

After deploying the template on Railway, follow these steps to configure your first AI provider and begin routing API requests.

1. Access the Dashboard

Open the public domain generated by Railway:

https://your-domain.up.railway.app

Sign in using the administrator credentials configured during deployment.

The dashboard provides access to provider management, model mappings, API keys, request logs, and usage statistics.

2. Add an AI Provider

Navigate to Providers and create a new provider configuration.

For example, to connect OpenAI:

SettingExample
ProviderOpenAI
Base URLhttps://api.openai.com/v1
ProtocolOpenAI
API KeyYour OpenAI API key

Save the configuration and use the connection test to verify connectivity.

Squirrel also supports Anthropic-compatible APIs, Azure OpenAI, and OpenAI-compatible inference servers such as Ollama and vLLM.

3. Configure Model Mapping

Navigate to Models and create a model mapping.

Model mappings define which upstream providers handle requests for a particular model name.

For example:

SettingExample
Model Namegpt-4.1-mini
ProviderOpenAI
Upstream Modelgpt-4.1-mini
Routing PriorityPrimary

You can associate multiple providers with a model and configure routing priorities or weights to support load balancing and failover.

4. Generate a Gateway API Key

Navigate to API Keys and generate a new gateway API key.

Use this key when connecting applications to Squirrel instead of exposing your upstream provider credentials directly.

Store the generated key securely because it grants access to the gateway according to its configured permissions.

5. Send Your First API Request

Squirrel exposes OpenAI-compatible endpoints, allowing existing applications to connect with minimal integration changes.

Replace the example domain, gateway API key, and model name with your deployment details.

Example using cURL:

curl -X POST "https://your-domain.up.railway.app/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1-mini",
    "messages": [
      {
        "role": "user",
        "content": "Hello from Railway!"
      }
    ]
  }'

Example using Python OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="https://your-domain.up.railway.app/v1",
    api_key="YOUR_GATEWAY_API_KEY"
)

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[
        {
            "role": "user",
            "content": "Hello from Railway!"
        }
    ]
)

print(response.choices[0].message.content)

Requests are forwarded to the configured upstream provider according to the model mapping and routing configuration.

6. Monitor Requests and Usage

After sending API requests, navigate to the dashboard to inspect:

  • Request history and response status
  • Model and provider usage
  • Token consumption
  • Request latency
  • Estimated API costs
  • Retry and failover activity

These metrics help troubleshoot provider errors, monitor AI spending, and optimize model routing.

Implementation Details

This Railway template uses a lightweight deployment architecture based on the official Squirrel Docker image.

ConfigurationValue
Docker Imageghcr.io/mylxsw/llm-gateway
Application Port8000
ArchitectureSingle container
DatabaseSQLite
Persistent Volume/data
RedisNot required
Custom Start CommandNot required
Public NetworkingHTTP/HTTPS via Railway

Data Persistence

Squirrel stores its database in the attached Railway volume, allowing provider configurations, model mappings, API keys, and operational data to survive container restarts and redeployments.

The SQLite deployment is designed for lightweight, single-instance hosting.

For larger production environments requiring concurrent replicas or more advanced database operations, PostgreSQL is a more suitable alternative.

Security Considerations

Protect the administration dashboard with strong credentials and restrict access to gateway API keys.

Upstream provider credentials should remain private and should never be embedded directly in frontend applications.

Request logging may contain sensitive prompts, responses, or application data. Review logging and retention settings before processing confidential information.

Backup and Recovery

Enable Railway Volume Backups to protect the persistent database.

Before updating the application:

  1. Create a backup of the persistent volume.
  2. Record the current Docker image version.
  3. Preserve administrator credentials and encryption material in secure storage.
  4. Update the application using a tested image version.
  5. Verify dashboard access, provider configurations, and API requests after deployment.

When restoring a database, use its original encryption key to ensure previously encrypted provider credentials remain accessible.

Avoid running multiple application replicas against the same SQLite database.

Why Deploy Squirrel LLM Gateway on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Squirrel LLM Gateway on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

Resources


Template Content

More templates in this category

View Template
Chat Chat
Chat Chat, your own unified chat and search to AI platform.

okisdev
117
View Template
Weaviate
Vector database with hybrid search and an API key-protected API

GitHub Locker
1
View Template
Möbius Personal AI OS
Self-hosted AI agent that builds and runs apps for you, with memory.

Tracklayer
1