Deploy promptfoo Eval Server + Auth Gateway

Deploy and Host promptfoo Eval Server + Auth Gateway on Railway

promptfoo is an open-source CLI and web UI for testing LLM prompts, models and agents: evals, side-by-side comparisons, assertions, and red teaming. This template runs the official promptfoo self-hosted server so your team and your CI can push eval results to one private dashboard.

About Hosting promptfoo Eval Server + Auth Gateway

The self-hosted promptfoo server has no login of its own: anyone who reaches it can read every eval and run evals with your provider keys. This template keeps the official ghcr.io/promptfoo/promptfoo:0.125.1 image on Railway's private network with no public domain, and puts a Caddy basic auth gateway in front of it. The gateway has two logins. The admin login opens the web UI and the full API. The upload login can only send results the way promptfoo share does, so the secret you put in CI cannot read your evals or start jobs. Evals live in SQLite on a Railway volume and survive redeploys. The server's own Share button, which would upload to the public promptfoo cloud, is turned off.

Common Use Cases

• Collect promptfoo eval results from GitHub Actions or any CI in one private dashboard
• Share eval and red team reports with teammates behind a password
• Compare prompts and models over time without sending results to a third party
• Run quick evals from the web UI eval creator with your own provider keys

Dependencies for promptfoo Eval Server + Auth Gateway Hosting

• Railway volume at /home/promptfoo/.promptfoo (included)
• No API keys needed to boot or to receive shared results. Provider keys are optional, only for running evals from the web UI

Deployment Dependencies

• promptfoo self-hosting guide
• Sharing results
• promptfoo on GitHub

Implementation Details

Services: promptfoo (private, port 3000, volume) and promptfoo-gateway (Caddy, public, port 8080).

First steps after deploy:

1. Open the promptfoo-gateway domain and sign in with ADMIN_USERNAME and the generated ADMIN_PASSWORD from that service's variables.
2. Copy PROMPTFOO_REMOTE_API_BASE_URL and PROMPTFOO_REMOTE_APP_BASE_URL from the gateway's variables into your CI secrets.
3. Run promptfoo eval then promptfoo share -y (or promptfoo eval --share). The printed link opens on your gateway.

# promptfoo-gateway
ADMIN_USERNAME=admin
ADMIN_PASSWORD=${{secret(32)}}
UPLOAD_USERNAME=ci
UPLOAD_PASSWORD=${{secret(32)}}
PROMPTFOO_REMOTE_API_BASE_URL=https://${{UPLOAD_USERNAME}}:${{UPLOAD_PASSWORD}}@${{RAILWAY_PUBLIC_DOMAIN}}
PROMPTFOO_REMOTE_APP_BASE_URL=https://${{RAILWAY_PUBLIC_DOMAIN}}

# promptfoo
API_PORT=3000
PROMPTFOO_CONFIG_DIR=/home/promptfoo/.promptfoo
RAILWAY_RUN_UID=0
PROMPTFOO_DISABLE_SHARING=true

The promptfoo CLI turns the user:password@ part of the API URL into a Basic auth header. Keep that URL in a masked secret. Upstream supports a single replica only (SQLite and in-memory jobs), so do not scale the promptfoo service out.

Why Deploy promptfoo Eval Server + Auth Gateway on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying promptfoo Eval Server + Auth Gateway on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.