Deploy Argilla | (Just Updated) Data Labeling & Annotation Platform for LLMs

Deploy and Host Argilla on Railway

Argilla is an open-source data labeling and annotation platform for LLM feedback. Teams use it to collect human preference data, rank and rate model outputs, label text for classification and extraction, and build instruction-tuning and RLHF datasets, with a Python SDK and a REST API for loading records and reading annotations back.

This deployment differs from the one-container quickstart image in four ways that matter once real annotators depend on it.

Your datasets survive a redeploy. The quickstart image keeps its database and search index inside one container filesystem. Here Postgres and Elasticsearch each have a volume, so annotations, datasets and users are still there after a redeploy or a restart.

The owner account exists before the first request. The owner user, its password and its API key are created at boot from PASSWORD and API_KEY, both generated for you. Nobody has to open the UI and register first, and anonymous registration is refused.

The image is pinned, not latest. Argilla 2.8.0 is pinned by digest, because the Argilla server migrates its schema forward on boot and an unrequested upgrade on redeploy is not a feature.

Elasticsearch is password protected on the private network. The search index is a private service with a generated ELASTIC_PASSWORD; only the Argilla server and worker are given the URL.

About Hosting Argilla

Argilla 2.x is a FastAPI server backed by PostgreSQL for metadata, Elasticsearch 8 for dataset search and filtering, and Redis plus a background worker for queued jobs. This template deploys all five pieces as five Railway services: argilla (the public web UI and API), worker, postgres, redis and elasticsearch.

The server waits for Postgres, Redis and Elasticsearch to answer before it starts, so first boot needs no ordering from you, and it honours Railway's injected $PORT. The healthcheck is GET /, which is served without credentials.

Agent contract

• Public URL: https://RAILWAY_PUBLIC_DOMAIN (the generated domain of the argilla service)
• Private URL: http://argilla.railway.internal:8080
• Credentials: the variables API_KEY and PASSWORD on the argilla service; the owner user name is argilla (add a USERNAME variable to change it before the first boot)
• Auth: send the API key as the header X-Argilla-Api-Key: $API_KEY. A write without the header answers 401, and a write with a wrong key answers 401. The web UI logs in with the user name argilla and PASSWORD.
• Health: GET / returns 200

#!/bin/sh
# needs: RAILWAY_PUBLIC_DOMAIN, API_KEY (variables of the argilla service); also curl.
B="https://$RAILWAY_PUBLIC_DOMAIN"
J="Content-Type: application/json"
N="agent-check-$(date +%s)"
curl -s -o /dev/null -w "health: %{http_code}\n" "$B/"
curl -s -o /dev/null -w "read with key: %{http_code}\n" -H "X-Argilla-Api-Key: $API_KEY" "$B/api/v1/me"
curl -s -o /dev/null -w "read without key: %{http_code}\n" "$B/api/v1/me"
curl -s -o /dev/null -w "write with key: %{http_code}\n" -X POST -H "$J" -H "X-Argilla-Api-Key: $API_KEY" -d "{\"name\":\"$N\"}" "$B/api/v1/workspaces"
curl -s -o /dev/null -w "write without key: %{http_code}\n" -X POST -H "$J" -d "{\"name\":\"$N-b\"}" "$B/api/v1/workspaces"
curl -s -o /dev/null -w "write with wrong key: %{http_code}\n" -X POST -H "$J" -H "X-Argilla-Api-Key: ${API_KEY}x" -d "{\"name\":\"$N-c\"}" "$B/api/v1/workspaces"
curl -s -o /dev/null -w "anonymous user create: %{http_code}\n" -X POST -H "$J" -d '{"username":"intruder","password":"intruder-password-1"}' "$B/api/v1/users"

A fresh deploy prints health: 200, read with key: 200, read without key: 401, write with key: 201, write without key: 401, write with wrong key: 401 and anonymous user create: 401.

Python SDK: rg.Argilla(api_url="https://RAILWAY_PUBLIC_DOMAIN", api_key=API_KEY).

Common Use Cases

• LLM preference and feedback data — rank, rate and compare model answers to build RLHF and DPO datasets
• Text classification and extraction labeling — multi-label, span and free-text annotation with review workflows for a team
• Evaluating RAG and agent outputs — send responses from your pipeline to the API and have reviewers score them in the UI

Dependencies for Argilla Hosting

• PostgreSQL — users, workspaces, datasets and annotations
• Elasticsearch 8 — search and filtering over dataset records
• Redis — queue for the background worker

Deployment Dependencies

• Argilla source, Apache-2.0
• Argilla documentation
• Railway images built for this template

After Deploying

1. Wait for the first deploy to finish; the server migrates the database and creates the owner account before it listens.
2. Open the public URL and log in as argilla with the generated PASSWORD from the argilla service's variables.
3. Create a workspace and add annotators under Users, or connect with the Python SDK using the API_KEY variable.

Why Deploy Argilla on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Argilla on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.