Deploy Deploy Langfuse Datasets and Experiments

Deploy and Host self hosted Langfuse Datasets (Open-Source LLM Datasets and Experiments) on Railway

This template runs a self-hosted Langfuse v3 stack for LLM datasets and experiments: versioned input and expected_output pairs, run from the UI or SDK before every prompt or model change. Built for teams that keep a golden set.

About Hosting Langfuse Datasets open-source software on Railway (self hosted Langfuse datasets template)

Your datasets, items and experiment runs sit in your own Postgres and ClickHouse instead of a vendor eval cloud.

One gotcha: restart langfuse-worker once after the first deploy finishes migrations.

Why Deploy Langfuse Datasets, the Braintrust alternative on Railway (Railway Free Trial)

Braintrust is a closed SaaS: your test sets live in their cloud and you pay per seat and per eval. Start on the $5 trial.

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Langfuse Datasets on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

Railway vs Other Hosting Providers and VPS for Langfuse Datasets self hosting

The difference is how much of the six-service stack you want to run yourself.

Either way you run the same MIT-licensed Langfuse 3 images, so datasets and runs stay portable.

Common Use Cases for hosted Langfuse Datasets

• Golden set of support questions: 40-80 real customer questions plus the answers you want, run before each prompt change so a fix for one case doesn't break five others.
• RAG regression tests: store expected_output as the reference answer, then score retrieval and generation separately to see which half drifted.
• Two models, one dataset: run the same items against two models through one LLM Connection and compare both runs side by side.
• Bad traces into items: add a failing production trace's input and your corrected expected_output from the UI, and the failure becomes permanent coverage.
• CI score gates: run the SDK experiment in a pipeline step and fail the build when average accuracy drops below your threshold.

Dependencies for Langfuse Datasets Docker hosted on Railway

Six services back each run. langfuse-web (langfuse/langfuse:3, public port 3000) serves UI and API; langfuse-worker (langfuse/langfuse-worker:3, port 3030) runs ingestion and evals. Web queues a Redis reference; the worker ingests into ClickHouse.

Deployment Dependencies for Managed Langfuse Datasets Service (LLM Evaluation Datasets)

DATABASE_URL, REDIS_CONNECTION_STRING (with ?family=0), CLICKHOUSE_URL, CLICKHOUSE_MIGRATION_URL, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, CLICKHOUSE_CLUSTER_ENABLED=false, LANGFUSE_S3_EVENT_UPLOAD_* and LANGFUSE_S3_MEDIA_UPLOAD_* refs, SALT, ENCRYPTION_KEY, NEXTAUTH_SECRET and NEXTAUTH_URL are set at deploy; secrets use ${{secret()}}.

Implementation Details for Langfuse Datasets (Using Langfuse official docker image)

Web runs langfuse/langfuse:3, so datasets and experiments match upstream docs. Health checks hit /api/public/health. Core is MIT with no usage limits.

How does Langfuse Datasets compare against other LLM dataset and experiment tools

All four run a test set through a task and score the output; they differ in where the data sits and how much config you write.

Langfuse vs Braintrust (Braintrust Alternative)

• Where runs live: Braintrust is closed SaaS. Self-hosted Langfuse keeps dataset items and run results in your own Postgres and ClickHouse.
• Cost shape: the OSS self-host has no usage limits; Braintrust charges per seat and per eval run, which adds up on big sets.
• Where Braintrust wins: a tighter experiment UI out of the box. Langfuse asks you to run six services.

Langfuse vs LangSmith (LangSmith Alternative)

• License and hosting: LangSmith is closed and self-hosts only on Enterprise. Langfuse core is MIT and runs on Railway today.
• Traces to datasets: both link runs to traces, so a failing item opens the real request. Langfuse needs no LangChain dependency.
• Where LangSmith wins: deeper LangChain and LangGraph wiring and more polished eval docs.

Langfuse vs Promptfoo (Promptfoo Alternative)

• Workflow style: Promptfoo is a YAML config CLI with red teaming built in. Langfuse is a UI plus SDK with datasets in a database and runs you compare side by side.
• Production cases: a bad trace becomes a Langfuse dataset item in two clicks; Promptfoo cases are files you edit by hand.
• Where Promptfoo wins: fastest zero-infra CI start and stronger adversarial presets.

Langfuse vs Arize Phoenix (Arize Phoenix Alternative)

• License: Phoenix is source-available under Elastic License 2.0. Langfuse core is MIT, though ee folders need a key.
• Data model: both store traces and evals, but Langfuse dataset runs are first-class objects you compare in the UI.
• Where Phoenix wins: OpenTelemetry-first tracing and a lighter single-container local setup.

How to use Langfuse Datasets (the OSS LLM experiment platform)?

1. Sign up on your Railway Langfuse domain and create a project.
2. Set LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST to your domain.
3. Create a dataset and items, then run an experiment:

from langfuse import Langfuse, Evaluation
lf = Langfuse()
lf.create_dataset(name="qa")
lf.create_dataset_item(dataset_name="qa", input="reset?",
    expected_output="Use the link.")

def task(*, item, **kwargs):
    return my_llm(item.input)

def accuracy(input, output, expected_output, **kw):
    return Evaluation(name="accuracy", value=float(output == expected_output))

ds = lf.get_dataset("qa")
ds.run_experiment(name="v2", task=task, evaluators=[accuracy])

4. Open Datasets in the UI to compare runs.

How to self host Langfuse Datasets on other VPS Services (Langfuse self hosting guide)

The docs call Docker Compose a local or single-VM setup. On a plain VPS you run all six pieces yourself.

Clone the Repository

Clone Langfuse:

git clone https://github.com/langfuse/langfuse.git
cd langfuse

Install Dependencies

Install Docker and Compose, then pull images:

docker compose pull

Configure Environment Variables

Edit docker-compose.yml and replace every value marked CHANGEME:

nano docker-compose.yml

Set NEXTAUTH_URL=https://your-domain, generate SALT, ENCRYPTION_KEY and NEXTAUTH_SECRET, and change the Postgres, Redis, ClickHouse and MinIO passwords.

Start the Langfuse Datasets Application

Start the stack and check the health endpoint:

docker compose up -d
curl https://your-domain/api/public/health

If migrations lag, restart the worker once. On Railway, the template provisions all of this and gives you a domain.

Official Pricing of Langfuse Datasets (Langfuse pricing)

Self-hosted Langfuse Datasets is free. The core repo is MIT licensed, has no usage limits or per-seat charge, and you only pay Railway, roughly $15-40/mo at low traffic for this six-service stack.

Langfuse Cloud is the paid option: Hobby is free at 50k units/mo, Core is $29/mo, Pro is $199/mo, Enterprise is $2,499/mo.

On your own stack, experiment runs cost nothing extra beyond LLM provider calls, which you pay for directly.

Langfuse Datasets cloud vs self hosted comparison (Pricing, features, costs, and more)

Dataset features are the same in both editions. What changes is who caps your experiment runs and where the scores live.

Monthly cost of self hosting Langfuse Datasets on Railway

The full six-service stack lands around $15-40/mo at low traffic, and the top of that range moves with ClickHouse disk and how many experiment runs you push through the worker. Cloud wins on effort.

System Requirements for Hosting Langfuse Datasets on a VPS

Upstream suggests 4 cores, 16 GiB RAM and 100 GiB disk for a VM install. Budget disk for ClickHouse rather than the web container, since traces and scores pile up there.

Frequently Asked Questions (FAQs)

How many items does a dataset need?

Enough to catch the failure you're worried about. Ten to thirty tricky items, each with an input and expected_output, beat a thousand random ones.

Can I add items from production traces?

Yes. Open a trace in the UI and add it to a dataset. A bad answer becomes a permanent regression case, the fastest loop there is.

Where are experiment scores stored?

Each experiment creates a dataset run, and scores land with the linked traces in ClickHouse, while items and runs sit in Postgres.

Can I run experiments without writing code?

Yes, with prompt experiments. Pick a prompt whose variables match the dataset input keys, choose a model from an LLM Connection, and Langfuse runs every item.

How do I compare two prompt versions?

Run the same dataset twice, once per prompt version, then compare the runs side by side in Datasets. Same items and evaluators, so the delta is the prompt.

Do experiments call my LLM provider, and who pays?

SDK experiments call whatever your task function calls, billed to your provider. UI prompt experiments run through langfuse-worker but still use your LLM Connection, so your provider bills you.