Deploy Agenta Prompt Management + Evals

Deploy and Host Agenta Prompt Management + Evals on Railway

Agenta is an open source LLMOps platform. Teams use it to version and deploy prompts, compare models side by side in a playground, run evaluations against test sets with built in or custom evaluators, collect human annotations, and trace LLM calls in production. Developers and product people work on the same prompts, and applications fetch the deployed version through the API or SDK.

About Hosting Agenta

Agenta's self host stack is not a single container. Upstream's compose runs a web app, a mobile web app, the API, a workflow services backend, an agent runner, stream and queue workers, a cron service, a migration job, SuperTokens for auth, PostgreSQL, Redis, a SeaweedFS object store and a Traefik proxy.

This template deploys that stack on Railway, pinned to v0.128.7, behind one public domain. A Caddy gateway does the path routing upstream's Traefik does (/api, /services, /m, and the web app). The API creates its three databases and runs migrations on boot, so there is no separate migration job to sequence, and the workers wait for it. Every secret is generated at deploy time and shared between services through references.

Common Use Cases

• Prompt management: version prompts, deploy them to environments, and fetch the live version from your app
• Prompt playground: compare prompts and models side by side before shipping a change
• Evaluations: run test sets through LLM as a judge, code and built in evaluators, or human review
• Observability: trace LLM calls (OpenTelemetry compatible) and turn production traces into test cases

Dependencies for Agenta Hosting

• PostgreSQL: included (three databases: core, tracing, SuperTokens)
• Redis: included (cache, streams and task queues)
• SeaweedFS: included (object store for agent files)
• SuperTokens: included (authentication)
• An LLM provider key: not needed to boot; add it per project in the UI

Deployment Dependencies

• Agenta self hosting quick start
• Agenta configuration reference
• Restrict sign ups and organization creation
• Agenta GitHub repository

Implementation Details

13 services, one public domain (the Agenta gateway).

# api (generated unless noted)
AGENTA_ACCESS_ALLOWED_OWNER_EMAILS=you@yourcompany.com   # required, you set it
AGENTA_AUTH_KEY=${{secret(64,"abcdef0123456789")}}
AGENTA_CRYPT_KEY=${{secret(64,"abcdef0123456789")}}
AGENTA_SERVICES_INTERNAL_KEY=${{secret(64,"abcdef0123456789")}}
AGENTA_RUNNER_TOKEN=${{secret(64,"abcdef0123456789")}}
POSTGRES_URI_CORE=postgresql+asyncpg://${{Postgres.POSTGRES_USER}}:${{Postgres.POSTGRES_PASSWORD}}@${{Postgres.RAILWAY_PRIVATE_DOMAIN}}:5432/agenta_oss_core
REDIS_URI=redis://default:${{Redis.REDIS_PASSWORD}}@${{Redis.RAILWAY_PRIVATE_DOMAIN}}:6379/0
SUPERTOKENS_URI_CORE=http://${{supertokens.RAILWAY_PRIVATE_DOMAIN}}:3567

# Agenta (gateway)
AGENTA_PUBLIC_URL=https://${{RAILWAY_PUBLIC_DOMAIN}}

The first deploy takes a few minutes while the API runs migrations on an empty database.

First Steps After Deploy

1. Open the Agenta service domain and sign up with the email you put in AGENTA_ACCESS_ALLOWED_OWNER_EMAILS. Only listed emails get an organization. Anyone else who signs up gets no organization and no access until you invite them, so a stranger who finds the URL cannot claim the instance. To block outside sign ups entirely, also set AGENTA_ACCESS_ALLOWED_DOMAINS on the api service.
2. Add a model provider key (OpenAI, Anthropic, and others) in your project settings.
3. Create a prompt in the playground, commit a variant, and deploy it to an environment.
4. Create an API key and fetch the deployed prompt from your app with the SDK (AGENTA_HOST = your domain) or https:///api with Authorization: ApiKey .
5. Invite teammates from Settings > Workspace > Members.

Notes: the agent runner runs without FUSE mounts on Railway (prompt management, evaluations and tracing are unaffected; use the Daytona sandbox provider for agents). Telemetry to Agenta is off. The stack needs the Hobby plan or above.

Why Deploy Agenta on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Agenta on Railway, you get the full open source LLMOps stack with generated secrets, managed PostgreSQL and Redis volumes, private networking between every service, health checks, and managed SSL on a single domain.