---
title: "Deploy promptfoo Eval Server + Auth Gateway"
description: "promptfoo eval server behind a login gateway, with a write-only CI upload"
category: "AI/ML"
url: https://railway.com/deploy/promptfoo-eval-server-auth-gateway
---

# Deploy promptfoo Eval Server + Auth Gateway

promptfoo eval server behind a login gateway, with a write-only CI upload

**[Deploy promptfoo Eval Server + Auth Gateway on Railway](https://railway.com/template/promptfoo-eval-server-auth-gateway)**

Machine-readable deploy manifest (JSON, validated by TemplateCI): https://railway.com/deploy/promptfoo-eval-server-auth-gateway/manifest.json

- **Creator:** Protemplate
- **Category:** AI/ML
- **Total deploys:** 2

## Template content

### promptfoo https://avatars.githubusercontent.com/u/137907881?v=4

- **Image:** ghcr.io/promptfoo/promptfoo:0.125.1
- **Health check:** /health

### promptfoo-gateway https://avatars.githubusercontent.com/u/137907881?v=4

- **Image:** caddy:2.11-alpine
- **Start command:** `sh -c 'echo IyEvYmluL3NoCiMgcHJvbXB0Zm9vLWdhdGV3YXkgc3RhcnQgc2NyaXB0IChjYWRkeToyLjExLWFscGluZSkuIEVtYmVkZGVkIGJhc2U2NCBpbiBzZXJpYWxpemVkLWNvbmZpZy5qc29uOgojICAgYmFzZTY0IDwgZ2F0ZXdheS1zdGFydC5zaCB8IHRyIC1kICdcbicKIyBUd28gY3JlZGVudGlhbHM6CiMgICBBRE1JTl8qICA6IGZ1bGwgYWNjZXNzICh3ZWIgVUkgKyB3aG9sZSBBUEkpLgojICAgVVBMT0FEXyogOiB3cml0ZSBvbmx5LiBNYXkgT05MWSBjYWxsIHRoZSB0d28gZW5kcG9pbnRzIGBwcm9tcHRmb28gc2hhcmVgIHVzZXMKIyAgICAgICAgICAgICAgKFBPU1QgL2FwaS9ldmFsIGFuZCBQT1NUIC9hcGkvZXZhbC88aWQ+L3Jlc3VsdHMpLiBQdXQgdGhpcyBvbmUgaW4gQ0kuCnNldCAtZQo6ICIke1BPUlQ6P1BPUlQgaXMgcmVxdWlyZWR9Igo6ICIke1VQU1RSRUFNOj9VUFNUUkVBTSBpcyByZXF1aXJlZH0iCjogIiR7QURNSU5fVVNFUk5BTUU6P0FETUlOX1VTRVJOQU1FIGlzIHJlcXVpcmVkfSIKOiAiJHtBRE1JTl9QQVNTV09SRDo/QURNSU5fUEFTU1dPUkQgaXMgcmVxdWlyZWR9Igo6ICIke1VQTE9BRF9VU0VSTkFNRTo/VVBMT0FEX1VTRVJOQU1FIGlzIHJlcXVpcmVkfSIKOiAiJHtVUExPQURfUEFTU1dPUkQ6P1VQTE9BRF9QQVNTV09SRCBpcyByZXF1aXJlZH0iCmlmIFsgIiRBRE1JTl9VU0VSTkFNRSIgPSAiJFVQTE9BRF9VU0VSTkFNRSIgXTsgdGhlbgogIGVjaG8gIkFETUlOX1VTRVJOQU1FIGFuZCBVUExPQURfVVNFUk5BTUUgbXVzdCBiZSBkaWZmZXJlbnQiID4mMgogIGV4aXQgMQpmaQpBRE1JTl9IQVNIPSIkKGNhZGR5IGhhc2gtcGFzc3dvcmQgLS1wbGFpbnRleHQgIiRBRE1JTl9QQVNTV09SRCIpIgpVUExPQURfSEFTSD0iJChjYWRkeSBoYXNoLXBhc3N3b3JkIC0tcGxhaW50ZXh0ICIkVVBMT0FEX1BBU1NXT1JEIikiCmNhdCA+IC9ldGMvY2FkZHkvQ2FkZHlmaWxlIDw8RU9GCnsKICBhdXRvX2h0dHBzIG9mZgogIGFkbWluIG9mZgogIHNlcnZlcnMgewogICAgcHJvdG9jb2xzIGgxIGgyYwogIH0KfQo6JHtQT1JUfSB7CiAgaGFuZGxlIC9nYXRld2F5LWhlYWx0aCB7CiAgICByZXNwb25kICJvayIgMjAwCiAgfQogIEB1cGxvYWQgewogICAgbWV0aG9kIFBPU1QKICAgIHBhdGhfcmVnZXhwIHVwbG9hZCBeL2FwaS9ldmFsKC9bXi9dKy9yZXN1bHRzKT8vP1wkCiAgfQogIGhhbmRsZSBAdXBsb2FkIHsKICAgIGJhc2ljX2F1dGggewogICAgICAke0FETUlOX1VTRVJOQU1FfSAke0FETUlOX0hBU0h9CiAgICAgICR7VVBMT0FEX1VTRVJOQU1FfSAke1VQTE9BRF9IQVNIfQogICAgfQogICAgcmV2ZXJzZV9wcm94eSAke1VQU1RSRUFNfQogIH0KICBoYW5kbGUgewogICAgYmFzaWNfYXV0aCB7CiAgICAgICR7QURNSU5fVVNFUk5BTUV9ICR7QURNSU5fSEFTSH0KICAgIH0KICAgIHJldmVyc2VfcHJveHkgJHtVUFNUUkVBTX0KICB9Cn0KRU9GCmV4ZWMgY2FkZHkgcnVuIC0tY29uZmlnIC9ldGMvY2FkZHkvQ2FkZHlmaWxlIC0tYWRhcHRlciBjYWRkeWZpbGUK | base64 -d > /tmp/gateway-start.sh && exec sh /tmp/gateway-start.sh'`
- **Health check:** /gateway-health
- **Public domain:** Yes

## Documentation

# Deploy and Host promptfoo Eval Server + Auth Gateway on Railway

promptfoo is an open-source CLI and web UI for testing LLM prompts, models and agents: evals, side-by-side comparisons, assertions, and red teaming. This template runs the official promptfoo self-hosted server so your team and your CI can push eval results to one private dashboard.

## About Hosting promptfoo Eval Server + Auth Gateway

The self-hosted promptfoo server has no login of its own: anyone who reaches it can read every eval and run evals with your provider keys. This template keeps the official `ghcr.io/promptfoo/promptfoo:0.125.1` image on Railway's private network with no public domain, and puts a Caddy basic auth gateway in front of it. The gateway has two logins. The admin login opens the web UI and the full API. The upload login can only send results the way `promptfoo share` does, so the secret you put in CI cannot read your evals or start jobs. Evals live in SQLite on a Railway volume and survive redeploys. The server's own Share button, which would upload to the public promptfoo cloud, is turned off.

## Common Use Cases

- Collect `promptfoo eval` results from GitHub Actions or any CI in one private dashboard
- Share eval and red team reports with teammates behind a password
- Compare prompts and models over time without sending results to a third party
- Run quick evals from the web UI eval creator with your own provider keys

## Dependencies for promptfoo Eval Server + Auth Gateway Hosting

- Railway volume at `/home/promptfoo/.promptfoo` (included)
- No API keys needed to boot or to receive shared results. Provider keys are optional, only for running evals from the web UI

### Deployment Dependencies

- [promptfoo self-hosting guide](https://www.promptfoo.dev/docs/usage/self-hosting/)
- [Sharing results](https://www.promptfoo.dev/docs/usage/sharing/)
- [promptfoo on GitHub](https://github.com/promptfoo/promptfoo)

### Implementation Details

Services: `promptfoo` (private, port 3000, volume) and `promptfoo-gateway` (Caddy, public, port 8080).

First steps after deploy:

1. Open the `promptfoo-gateway` domain and sign in with `ADMIN_USERNAME` and the generated `ADMIN_PASSWORD` from that service's variables.
2. Copy `PROMPTFOO_REMOTE_API_BASE_URL` and `PROMPTFOO_REMOTE_APP_BASE_URL` from the gateway's variables into your CI secrets.
3. Run `promptfoo eval` then `promptfoo share -y` (or `promptfoo eval --share`). The printed link opens on your gateway.

```env
# promptfoo-gateway
ADMIN_USERNAME=admin
ADMIN_PASSWORD=${{secret(32)}}
UPLOAD_USERNAME=ci
UPLOAD_PASSWORD=${{secret(32)}}
PROMPTFOO_REMOTE_API_BASE_URL=https://${{UPLOAD_USERNAME}}:${{UPLOAD_PASSWORD}}@${{RAILWAY_PUBLIC_DOMAIN}}
PROMPTFOO_REMOTE_APP_BASE_URL=https://${{RAILWAY_PUBLIC_DOMAIN}}

# promptfoo
API_PORT=3000
PROMPTFOO_CONFIG_DIR=/home/promptfoo/.promptfoo
RAILWAY_RUN_UID=0
PROMPTFOO_DISABLE_SHARING=true
```

The promptfoo CLI turns the `user:password@` part of the API URL into a Basic auth header. Keep that URL in a masked secret. Upstream supports a single replica only (SQLite and in-memory jobs), so do not scale the promptfoo service out.

## Why Deploy promptfoo Eval Server + Auth Gateway on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying promptfoo Eval Server + Auth Gateway on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.


## Similar templates

- [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform.
- [Weaviate](https://railway.com/deploy/weaviate-1) — Vector database with hybrid search and an API key-protected API
- [Möbius v0.1 Personal AI OS](https://railway.com/deploy/mobius-v01-personal-ai-os) — Self-hosted AI agent that builds and runs apps for you, with memory.

Open this page in a browser: https://railway.com/deploy/promptfoo-eval-server-auth-gateway