Deploy Promptfoo — Self-Hosted LLM Eval Dashboard

Deploy and Host Promptfoo on Railway

Promptfoo is a popular open-source tool for testing, evaluating, and red-teaming LLM apps — used by teams to catch prompt regressions, compare models, and probe applications for vulnerabilities. This template deploys the self-hosted Promptfoo server: a private web dashboard where your team's evaluation results live on your own infrastructure instead of the public cloud. Run evals locally or in CI, then share the results to your Railway instance — keeping sensitive prompts, outputs, and red-team findings entirely under your control.

What This Template Deploys

A single lightweight service. A persistent volume at /home/promptfoo/.promptfoo holds the SQLite database (promptfoo.db) with your evaluation history, so results survive redeploys. Railway provides a public HTTPS URL for the dashboard.

About Hosting

This template hosts Promptfoo's self-hosted results dashboard — knowing exactly what it's for keeps expectations right, and this template handles the setup.

It's a private results dashboard and sharing target — not where you run evals. The main workflow stays the same: you run promptfoo eval locally or in your CI pipeline. What this server changes is where the results go — instead of publishing to Promptfoo's public cloud with promptfoo eval --share, you point sharing at your Railway instance, so your team browses, compares, and reviews eval history on infrastructure you own. This is the reason to self-host: keeping sensitive prompts and outputs private.

Point your local Promptfoo at this instance. After deploy, set your local environment to share to your Railway URL: PROMPTFOO_REMOTE_API_BASE_URL=https://your-instance.railway.app and PROMPTFOO_SHARE_STORE_TYPE=database. Then promptfoo eval --share publishes results to your private dashboard instead of the public site. You can also run evals directly from the web UI if you add provider keys.

Persist the volume — it holds all your eval history. The SQLite database at /home/promptfoo/.promptfoo stores every evaluation you've shared. Without the mounted volume, that history is wiped on redeploy. This template mounts it so your results persist.

Best for individuals and small teams — by design. Promptfoo's self-hosted server is intended for individual or small-team use: SQLite, no horizontal scaling, no multi-team access control or SSO. Run a single instance (multiple replicas can't share the SQLite database and will error). For organization-wide or high-scale needs, Promptfoo offers an Enterprise platform — this template fits a team wanting a private, self-hosted results hub.

Add provider keys for in-UI evals. Optionally set OPENAI_API_KEY, ANTHROPIC_API_KEY, or others so users can create and run evals directly from the web UI, bringing your own keys. Without them, the instance still serves as the results dashboard for CLI/CI evals.

Red-teaming and regression testing. Use Promptfoo to run adversarial tests against your LLM app and review attack results and failure modes in the dashboard, and to catch prompt regressions automatically in CI — with all findings stored privately on your instance.

Typical cost: ~$5/month on Railway — Promptfoo is lightweight, since the heavy compute happens on the LLM provider side. It's MIT-licensed and free.

How It Compares

Promptfoo's public cloud is easy but stores your eval results on their servers. Manual testing keeps everything local but offers no shared dashboard or history. Enterprise eval platforms scale but cost and require sales. This self-hosted instance's edge is a private, team-shareable results dashboard for your evals and red-team findings — on infrastructure you own, at flat cost — ideal for a small team that wants privacy without enterprise overhead.

Deploy in Under 5 Minutes

1. Click Deploy on Railway — Promptfoo builds with a volume for eval history (~2 minutes)
2. Confirm the volume is mounted at /home/promptfoo/.promptfoo
3. Open your Railway URL to access the Promptfoo web dashboard
4. Locally, set PROMPTFOO_REMOTE_API_BASE_URL to your Railway URL and PROMPTFOO_SHARE_STORE_TYPE=database
5. Run promptfoo eval --share — results publish to your private dashboard

Optionally add provider API keys so users can run evals directly from the web UI.

Common Use Cases

• Private eval dashboard — a team hub for LLM evaluation results on your own infrastructure
• CI/CD eval aggregation — collect automated prompt-regression results from CI in one place
• Prompt regression testing — catch quality drops across prompt and model changes over time
• Model comparison — compare GPT, Claude, Gemini, and others, with results kept in-house

Configuration

Persist the volume — it's your eval history. The SQLite database lives at /home/promptfoo/.promptfoo; without the volume, a redeploy wipes it.

Point local Promptfoo here to share. Set PROMPTFOO_REMOTE_API_BASE_URL to your Railway URL and PROMPTFOO_SHARE_STORE_TYPE=database, then promptfoo eval --share publishes to your private instance. Run a single instance only.

Dependencies for Promptfoo Hosting

• Railway account — ~$5/month for the lightweight service
• A persistent volume at /home/promptfoo/.promptfoo for eval history (included)
• The Promptfoo CLI installed locally or in CI (npm install -g promptfoo) to run evals
• Optional: LLM provider API keys to run evals directly from the web UI

Deployment Dependencies

• Promptfoo GitHub Repository
• Promptfoo Documentation
• Promptfoo Self-Hosting Guide
• Railway Volumes Documentation

Implementation Details

The template runs the official ghcr.io/promptfoo/promptfoo image as a single Express server on port 3000, serving the web UI and API, with a persistent volume mounted at /home/promptfoo/.promptfoo to retain the SQLite database (promptfoo.db) of eval history across redeploys. PROMPTFOO_DISABLE_TELEMETRY=1 is set, and provider API keys can optionally be added so evals run from the web UI.

The intended workflow is that evaluations run locally or in CI via the Promptfoo CLI, and results are shared to this instance by setting PROMPTFOO_REMOTE_API_BASE_URL to the Railway URL and PROMPTFOO_SHARE_STORE_TYPE=database, then running promptfoo eval --share — publishing to the private instance instead of Promptfoo's public cloud. This keeps prompts, outputs, and red-team findings on infrastructure you control.

Per Promptfoo's guidance, the self-hosted server targets individual or small-team use: it relies on SQLite, offers no multi-team RBAC or SSO, and does not support horizontal scaling — jobs live in server memory and replicas cannot share the SQLite database, so only a single instance should run. Organizations needing scale should use Promptfoo Enterprise. The volume holding promptfoo.db is the component to back up.

Frequently Asked Questions

Do I run evals on this server or locally? Primarily locally or in CI with the Promptfoo CLI. This instance is the private dashboard where you share and review results, though you can also run evals from the web UI if you add provider keys.

How do I send results to my instance? Set PROMPTFOO_REMOTE_API_BASE_URL to your Railway URL and PROMPTFOO_SHARE_STORE_TYPE=database in your local environment, then run promptfoo eval --share. Results publish to your private dashboard.

Does my eval history persist? Yes — the SQLite database at /home/promptfoo/.promptfoo holds your history and is mounted on a persistent volume, so it survives redeploys. Back up the volume to preserve results.

Is this suitable for a large organization? It's designed for individuals and small teams — no multi-team RBAC, SSO, or horizontal scaling. For organization-wide use, Promptfoo offers an Enterprise platform.

Can I do red-teaming with it? Yes — run Promptfoo's adversarial and vulnerability tests against your LLM app, and review the attack results and failure modes in the dashboard, kept private on your instance.

Why Deploy Promptfoo on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Promptfoo on Railway you get a private, self-hosted eval dashboard with the setup handled — the official image, a persistent volume for your eval history, and automatic HTTPS. Keep your LLM testing and red-team results on infrastructure you own, ideal for individuals and small teams.