Deploy Langfuse Evals

Deploy and Host self hosted Langfuse Evals (Open-Source LLM Evaluation) on Railway

This template runs Langfuse Evals on Railway: judge prompts with {{input}}, {{output}} and {{ground_truth}} return numeric, categorical or boolean scores, executed by langfuse-worker on live observations or dataset experiments, using a model from Project Settings.

About Hosting Langfuse Evals open-source software on Railway (self hosted Langfuse evaluation template)

This template runs the full Langfuse v3 stack: web, worker, Postgres, Redis, ClickHouse, and a Railway Storage Bucket. Your traces, scores, and judge prompts stay in your own Railway project, not on Langfuse Cloud.

Why Deploy Langfuse Evals, the Braintrust alternative on Railway (Railway Free Trial)

Braintrust is closed-source SaaS, so eval history and judge prompts live on someone else's infra. Self-hosting Langfuse Evals keeps datasets, judges and scores under your control with MIT, and Railway's $5 trial covers a test drive; you only pay provider charges for judge calls.

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Langfuse Evals on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.

Railway vs Other Hosting Providers and VPS for Langfuse Evals self hosting

Railway ships the full managed stack in one template, while a raw VPS leaves every service to you.

Common Use Cases for hosted Langfuse Evals

• Live scoring: Set a rule on root LLM calls with a sampling rate; a judge scores helpfulness or correctness to spot drift without blocking traffic.
• Regression tests: Build a dataset from tricky inputs with outputs, run judges on old and new prompt versions, and compare before you promote.
• RAG faithfulness: Point an evaluator at retrieval spans with {{input}} as question and {{output}} as chunks, boolean judge flags hallucinated citations.
• Agent tool calls: Target tool-call observations, use a categorical judge to label whether the agent picked the right tool and arguments, and track wrong-tool rates.
• Calibration: Label a sample of outputs via annotation queues, then compare those labels with judge scores in Score Analytics to tune prompts or swap models.

Dependencies for Langfuse Evals Docker hosted on Railway

This template runs six services: langfuse-web, langfuse-worker, Postgres, Redis, ClickHouse 24.8, and a Railway Storage Bucket. Secrets generate per deploy, so there's no config step. Evaluators run inside the worker, which needs outbound access to your LLM provider.

Deployment Dependencies for Managed Langfuse Evals Service (LLM Evaluation)

• langfuse-web (langfuse/langfuse:3, public, port 3000) serves UI/API.
• langfuse-worker (langfuse/langfuse-worker:3, port 3030) does async ingestion and evals.
• Postgres stores transactional data; Redis queues/caches; ClickHouse stores traces/observations/scores on a /var/lib/clickhouse volume.
• Railway Storage Bucket replaces MinIO via LANGFUSE_S3_EVENT_UPLOAD_* and LANGFUSE_S3_MEDIA_UPLOAD_*.
• Env vars include DATABASE_URL, REDIS_CONNECTION_STRING (?family=0), CLICKHOUSE_URL, CLICKHOUSE_MIGRATION_URL, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, CLICKHOUSE_CLUSTER_ENABLED=false, SALT, ENCRYPTION_KEY (64 hex), NEXTAUTH_SECRET, NEXTAUTH_URL.

Implementation Details for Langfuse Evals (Using Langfuse official docker image)

Web writes event batches to S3 and queues a Redis reference; worker ingests into ClickHouse. Add a provider key under Project Settings, LLM Connections; the judge model must support structured output. MIT covers evals and datasets with no license key.

How does Langfuse Evals compare against other LLM evaluation platforms

Langfuse vs Braintrust (Braintrust Alternative)

• License and hosting: MIT, runs on your own Railway stack; Braintrust is closed SaaS, so eval data stays in their cloud.
• Cost: Self-hosted pays only judge model tokens; Braintrust adds platform pricing on top.
• Braintrust wins: Easier managed start if you don't want to operate ClickHouse and Redis.

Langfuse vs LangSmith (LangSmith Alternative)

• Open source: Langfuse evals are in the MIT repo; LangSmith self-hosting is Enterprise only.
• Lock-in: Langfuse is framework-agnostic; LangSmith is deepest inside LangChain.
• LangSmith wins: Better turnkey tracing for LangGraph-heavy teams.

Langfuse vs Arize Phoenix (Arize Phoenix Alternative)

• License: Phoenix is Elastic 2.0, not OSI open source; Langfuse is MIT.
• Execution: Langfuse runs judges in its worker automatically; Phoenix leans on an evals library you run from code.
• Phoenix wins: Stronger OTel tracing layer if observability comes first.

Langfuse vs Promptfoo (Promptfoo Alternative)

• Model: Promptfoo is CLI and CI; Langfuse scores live production observations continuously.
• Persistence: Langfuse stores scores and judge reasoning in UI; Promptfoo outputs to terminal or files by default.
• Promptfoo wins: Lighter for pre-deploy prompt tests in a repo.

How to use Langfuse Evals (the OSS LLM evaluation platform)?

1. Sign up on your Railway domain, create a project, and copy the API keys.
2. Send traces from your SDK with LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST set to your Railway domain.
3. Add an LLM Connection under Project Settings.
4. Create an LLM-as-a-Judge evaluator from the template gallery, then map {{input}} and {{output}}.
5. Test it on sample observations and check the scores before saving anything.
6. Save it with a rule: pick observation filters, trace filters, and a sampling rate.
7. Build a dataset and run an experiment before your next prompt change.
8. Filter traces to the langfuse-llm-as-a-judge environment to read judge reasoning.

How to self host Langfuse Evals on other VPS Services (Langfuse self hosting guide)

Clone the Repository

git clone https://github.com/langfuse/langfuse.git
cd langfuse

Install Dependencies

Install Docker and Docker Compose. The repo's docker-compose.yml already starts Postgres, Redis, ClickHouse and MinIO.

Configure Environment Variables

Replace every secret marked CHANGEME in the compose file, including SALT, ENCRYPTION_KEY, NEXTAUTH_SECRET and the database passwords, and set NEXTAUTH_URL to your domain. MinIO covers event and media uploads, or point the LANGFUSE_S3_* vars at your own bucket.

Start the Langfuse Evals Application

Run docker compose up -d, then restart the worker once after the web service finishes migrations. Sign in at your-domain, add an LLM Connection and create your first judge.

Official Pricing of Langfuse Evals (Langfuse pricing)

Self-hosting the MIT-licensed Langfuse Evals stack is free with no usage limits; you pay only Railway infrastructure and judge model calls on your own provider bill.

Langfuse Evals cloud vs self hosted comparison (Pricing, features, costs, and more)

Self-hosting flips the unit math: you pay Railway for the stack and your LLM provider for judge calls, not Langfuse for trace or score units. The OSS build runs datasets, experiments, LLM-as-a-judge and human annotation with no license key.

Monthly cost of self hosting Langfuse Evals on Railway

Expect roughly $15 to $40 per month for this six-service stack with ClickHouse at low traffic, plus provider charges for every judge call, typically 1 to 10 cents depending on model and input size. The Railway $5 trial covers early testing, and there is no Langfuse usage cap on self-hosted OSS.

System Requirements for Hosting Langfuse Evals on a VPS

Official self-host guidance suggests 4 cores, 16 GiB RAM and 100 GiB disk for a single VM. That suits Docker Compose for local or test setups, though production deployments need separate Postgres, Redis and ClickHouse services with persistent storage, which this Railway template already provisions.

Frequently Asked Questions (FAQs)

Who pays for judge calls?

Judge calls are billed by your model provider, not self-hosted Langfuse.

Which models work as judges?

Any model in LLM Connections that supports structured output with a JSON schema, including OpenAI, Anthropic, Azure, Bedrock, Google, and OpenAI-compatible endpoints.

Why does saving an evaluator fail with "Model configuration not valid for evaluation"?

Langfuse sends one test request on save. If the model or a proxy lacks structured output, the save fails; switch to a model that accepts JSON schema.

How do I judge only the final answer and not every span?

Use the "Is Root Observation" filter and target the observation that records overall input and output. An evaluator sees only that observation, not its children.

Can I add my own code-based scores?

Yes. Send custom scores through the SDK or public API, or collect user feedback in the UI. Deterministic checks sit next to judge scores, and Score Analytics compares them with human labels.

Are evals included in the free self-hosted build?

Yes. Evaluators, datasets, experiments, and human annotation are MIT-licensed, so no license key is needed. Audit logs, SCIM, and protected prompt labels require an enterprise license.