
Deploy Squirrel
Unified LLM API gateway with smart routing and failover.
Squirrel
Just deployed
/data
Deploy and Host Squirrel LLM Gateway on Railway
Squirrel LLM Gateway is an open-source, self-hosted AI API gateway that unifies access to multiple Large Language Model (LLM) providers. It offers intelligent model routing, automatic failover, load balancing, API key management, request logging, and cost analytics. Squirrel supports OpenAI-compatible and Anthropic-compatible APIs through a centralized management dashboard.
About Hosting Squirrel LLM Gateway
Hosting Squirrel LLM Gateway on Railway provides a centralized API endpoint for connecting applications to multiple AI providers without managing complex infrastructure.
This template uses the official Docker image with a lightweight, single-service architecture powered by SQLite. Railway handles container deployment, HTTPS networking, and persistent storage through an attached volume.
The gateway runs on port 8000 and includes a web-based administration dashboard for managing providers, model mappings, API keys, routing policies, and usage analytics.
Once deployed, applications can connect to Squirrel using OpenAI-compatible SDKs and API endpoints, while the gateway handles upstream provider communication, retries, and request routing.
Common Use Cases
- Unified AI API Gateway: Connect multiple LLM providers through a single API endpoint without maintaining separate integrations.
- Intelligent Model Routing: Route requests based on model names, priorities, weights, costs, and configurable routing rules.
- Automatic Provider Failover: Improve application reliability through automatic retries and fallback providers.
- API Key Management: Issue gateway API keys while keeping upstream provider credentials centralized.
- AI Usage Monitoring: Track requests, token consumption, latency, and estimated API costs through a management dashboard.
- Multi-Provider AI Applications: Provide a consistent API interface for AI agents, chatbots, automation workflows, and backend applications.
Dependencies for Squirrel LLM Gateway Hosting
- Squirrel Docker Image: Official container image
ghcr.io/mylxsw/llm-gateway, exposing HTTP port8000. - SQLite Database: Built-in database for storing providers, model mappings, API keys, and application configuration.
- Persistent Storage: One Railway volume mounted at
/datato retain SQLite data across restarts and redeployments. - LLM Provider: At least one external or self-hosted AI provider with an accessible API endpoint and valid credentials.
- Railway Service: A single container service with public HTTP networking enabled.
- Redis: Not required for this template.
Squirrel LLM Gateway vs. Alternatives
Squirrel focuses on unified LLM access, intelligent routing, and observability with a relatively lightweight deployment architecture.
| Feature | Squirrel LLM Gateway | LiteLLM Proxy | New API |
|---|---|---|---|
| Primary Focus | Intelligent LLM routing | Multi-provider AI gateway | AI API management |
| Self-Hosted | ✅ | ✅ | ✅ |
| OpenAI-Compatible API | ✅ | ✅ | ✅ |
| Multiple LLM Providers | ✅ | ✅ | ✅ |
| Web Management Dashboard | ✅ | ✅ | ✅ |
| API Key Management | ✅ | ✅ | ✅ |
| Load Balancing | ✅ | ✅ | ✅ |
| Automatic Failover | ✅ | ✅ | ⚠️ Configuration-dependent |
| Usage & Cost Analytics | ✅ | ✅ | ✅ |
| SQLite Support | ✅ | ✅ | ✅ |
| Single-Container Deployment | ✅ | ✅ | ✅ |
| Advanced Routing Rules | ✅ | ✅ | ⚠️ |
| Deployment Complexity | Low–Medium | Medium | Low–Medium |
Choose Squirrel LLM Gateway for intelligent routing, failover, and detailed AI request observability through a unified gateway.
Choose LiteLLM Proxy for extensive model provider compatibility, AI infrastructure integrations, and advanced gateway functionality.
Choose New API for centralized API distribution, provider channel management, token administration, and usage tracking.
How to Use Squirrel LLM Gateway
After deploying the template on Railway, follow these steps to configure your first AI provider and begin routing API requests.
1. Access the Dashboard
Open the public domain generated by Railway:
https://your-domain.up.railway.app
Sign in using the administrator credentials configured during deployment.
The dashboard provides access to provider management, model mappings, API keys, request logs, and usage statistics.
2. Add an AI Provider
Navigate to Providers and create a new provider configuration.
For example, to connect OpenAI:
| Setting | Example |
|---|---|
| Provider | OpenAI |
| Base URL | https://api.openai.com/v1 |
| Protocol | OpenAI |
| API Key | Your OpenAI API key |
Save the configuration and use the connection test to verify connectivity.
Squirrel also supports Anthropic-compatible APIs, Azure OpenAI, and OpenAI-compatible inference servers such as Ollama and vLLM.
3. Configure Model Mapping
Navigate to Models and create a model mapping.
Model mappings define which upstream providers handle requests for a particular model name.
For example:
| Setting | Example |
|---|---|
| Model Name | gpt-4.1-mini |
| Provider | OpenAI |
| Upstream Model | gpt-4.1-mini |
| Routing Priority | Primary |
You can associate multiple providers with a model and configure routing priorities or weights to support load balancing and failover.
4. Generate a Gateway API Key
Navigate to API Keys and generate a new gateway API key.
Use this key when connecting applications to Squirrel instead of exposing your upstream provider credentials directly.
Store the generated key securely because it grants access to the gateway according to its configured permissions.
5. Send Your First API Request
Squirrel exposes OpenAI-compatible endpoints, allowing existing applications to connect with minimal integration changes.
Replace the example domain, gateway API key, and model name with your deployment details.
Example using cURL:
curl -X POST "https://your-domain.up.railway.app/v1/chat/completions" \
-H "Authorization: Bearer YOUR_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4.1-mini",
"messages": [
{
"role": "user",
"content": "Hello from Railway!"
}
]
}'
Example using Python OpenAI SDK:
from openai import OpenAI
client = OpenAI(
base_url="https://your-domain.up.railway.app/v1",
api_key="YOUR_GATEWAY_API_KEY"
)
response = client.chat.completions.create(
model="gpt-4.1-mini",
messages=[
{
"role": "user",
"content": "Hello from Railway!"
}
]
)
print(response.choices[0].message.content)
Requests are forwarded to the configured upstream provider according to the model mapping and routing configuration.
6. Monitor Requests and Usage
After sending API requests, navigate to the dashboard to inspect:
- Request history and response status
- Model and provider usage
- Token consumption
- Request latency
- Estimated API costs
- Retry and failover activity
These metrics help troubleshoot provider errors, monitor AI spending, and optimize model routing.
Implementation Details
This Railway template uses a lightweight deployment architecture based on the official Squirrel Docker image.
| Configuration | Value |
|---|---|
| Docker Image | ghcr.io/mylxsw/llm-gateway |
| Application Port | 8000 |
| Architecture | Single container |
| Database | SQLite |
| Persistent Volume | /data |
| Redis | Not required |
| Custom Start Command | Not required |
| Public Networking | HTTP/HTTPS via Railway |
Data Persistence
Squirrel stores its database in the attached Railway volume, allowing provider configurations, model mappings, API keys, and operational data to survive container restarts and redeployments.
The SQLite deployment is designed for lightweight, single-instance hosting.
For larger production environments requiring concurrent replicas or more advanced database operations, PostgreSQL is a more suitable alternative.
Security Considerations
Protect the administration dashboard with strong credentials and restrict access to gateway API keys.
Upstream provider credentials should remain private and should never be embedded directly in frontend applications.
Request logging may contain sensitive prompts, responses, or application data. Review logging and retention settings before processing confidential information.
Backup and Recovery
Enable Railway Volume Backups to protect the persistent database.
Before updating the application:
- Create a backup of the persistent volume.
- Record the current Docker image version.
- Preserve administrator credentials and encryption material in secure storage.
- Update the application using a tested image version.
- Verify dashboard access, provider configurations, and API requests after deployment.
When restoring a database, use its original encryption key to ensure previously encrypted provider credentials remain accessible.
Avoid running multiple application replicas against the same SQLite database.
Why Deploy Squirrel LLM Gateway on Railway?
Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.
By deploying Squirrel LLM Gateway on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.
Resources
- Official Repository: https://github.com/mylxsw/llm-gateway
- Official Website: https://squirrel.gulu.ai/
- API Documentation: https://github.com/mylxsw/llm-gateway/blob/main/docs/api.md
- Architecture Documentation: https://github.com/mylxsw/llm-gateway/blob/main/docs/architecture.md
- Railway Documentation: https://docs.railway.com/
Template Content

