Deploy Dagster

Deploy and Host Dagster on Railway

Dagster is an open-source data orchestrator built around software-defined assets. Rather than describing a pipeline as a sequence of tasks, you declare the tables, files and models your code produces; Dagster works out what to run, records every materialization, and tracks lineage between them. Data teams use it to schedule dbt projects, keep warehouse tables fresh, and get a catalog that explains where a number came from.

Self-host Dagster on Railway with the production split its own documentation recommends: a webserver for the UI and GraphQL API, a daemon for schedules, sensors and the run queue, and a gRPC code server that loads your Python and executes runs. Deploy Dagster with Postgres wired up for run storage and the event log, a bucket for step logs and asset values, and an authenticating gateway in front — the open-source webserver has no login, and anything that reaches it can launch and terminate runs.

Getting Started with Dagster on Railway

Open the deployed URL and the browser asks for a username and password — the DAGSTER_AUTH_USER and DAGSTER_AUTH_PASSWORD you set at deploy time. There is no in-app signup, so change the password by editing that variable. Past it you land on the UI with a code location named railway already loaded.

Click Lineage to see the asset graph. The template ships a self-contained example — synthetic storefront orders rolled into daily revenue, a product ranking and a summary report — so there is something real to run before you write code. Press Materialize all: the daemon queues the run and launches it into the code server, each asset executes in its own subprocess, and within about ten seconds every node turns green and the check on raw_orders reports 1/1 passed. Open the run from Runs for the step timeline, the event log and each step's captured stdout, streamed to the bucket rather than local disk. That single run exercises Postgres, the bucket, the queue and the code server at once.

To run your own code, replace project/railway_demo/ with your project, add its dependencies to requirements.txt, and point DAGSTER_MODULE_NAME at your Definitions module; only the code server rebuilds. Schedules ship switched off — enable them from Automation.

About Hosting Dagster

Dagster inverts the usual orchestration model. A task scheduler knows only that step B follows step A; Dagster knows daily_revenue is built from raw_orders, so it can say which downstream tables are stale, re-run only what changed, and attach metadata to each materialization.

• Software-defined assets with lineage across Python, dbt, Spark and SQL
• Asset checks that assert data quality and surface failures in the catalog
• Schedules, sensors and declarative automation from one daemon process
• Backfills and partitions for reprocessing date ranges without custom scripts
• A run queue with concurrency limits, and event logs queryable via GraphQL

The deployment splits into four roles because Dagster expects them separate. The webserver serves the UI and GraphQL API and holds no state. The daemon owns everything time-based: ticking schedules, evaluating sensors, dequeuing runs and monitoring crashed ones. The code server isolates your project, so a broken import shows as a code location error rather than taking the instance down; run workers spawn inside its container. Postgres holds run history, the event log and schedule state; the bucket holds step stdout and the values assets pass between each other, since every run worker is a separate process with its own filesystem.

Why Deploy Dagster on Railway

Railway removes the infrastructure work self-hosting an orchestrator involves:

• Postgres and object storage are provisioned and wired up for you
• The four services talk over a private network; only the gateway is public
• Health checks probe the daemon heartbeat and the gRPC server, not just a port
• Pushing to the source repository rebuilds and redeploys your code location

Common Use Cases

• Orchestrating dbt — models on a schedule, with lineage, freshness and tests in one catalog
• Warehouse and ELT pipelines — pull APIs and databases into Postgres, Snowflake or BigQuery, with retries and backfills
• ML pipelines — version feature datasets as assets, retrain when inputs change
• Scheduled reporting — rebuild aggregates nightly, alerting when a check fails

Dependencies for Dagster

• dagster-webserver, dagster-daemon, dagster-code, dagster-gateway — all built from gridalpha/dagster-railway on python:3.12-slim, serving Dagster 1.13.19; the gateway role runs Caddy and holds the public domain
• Postgres — managed Postgres for run storage, the event log and schedules
• Object storage bucket — compute logs and the S3 pickle IO manager

Environment Variables Reference

| DAGSTER_S3_BUCKET / _ENDPOINT / _REGION | Bucket for step logs and asset values |

Deployment Dependencies

• Source: gridalpha/dagster-railway; upstream dagster-io/dagster, docs at docs.dagster.io
• Python packages: dagster, dagster-webserver, dagster-postgres, dagster-aws

Hardware Requirements for Self-Hosting Dagster

Sizing depends on what your assets do; the orchestration layer is light. The figures below cover the four services combined, excluding your workload.

Scale the code server first: run workers are its child processes, so raising DAGSTER_MAX_CONCURRENT_RUNS means sizing it for peak concurrency.

Self-Hosting Dagster

Dagster publishes no ready-made server image — you build one from the Python packages. A local install is three commands, the last starting a dev UI on port 3000:

pip install dagster dagster-webserver dagster-postgres dagster-aws
export DAGSTER_HOME=$PWD/dagster_home && mkdir -p "$DAGSTER_HOME"
dagster dev -m my_project.definitions

A production install splits the processes and points them at Postgres, via dagster.yaml in $DAGSTER_HOME:

storage:
  postgres:
    postgres_url:
      env: DAGSTER_PG_URL
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator

Then run three processes, each in its own container: dagster code-server start -h 0.0.0.0 -p 4000 -m my_project.definitions, dagster-webserver -h 0.0.0.0 -p 3000 -w workspace.yaml, and dagster-daemon run. Keep all four packages on the same release — they exchange serialised objects over gRPC, and a mismatch fails at run time.

Is Dagster Free?

Dagster is free and open source under the Apache 2.0 license, with no feature gating, seat limits or asset caps in the self-hosted build. Dagster+ adds a hosted control plane, branch deployments, alerting and access control over the same engine. Self-hosting costs only the compute, Postgres and storage the deployment uses.

FAQ

What is Dagster?

An open-source orchestrator for data and machine-learning pipelines. It is asset-centric: you declare the tables and models your code produces, and Dagster schedules them, tracks materializations and records lineage.

What does this Railway template deploy?

A Dagster webserver, daemon, gRPC code server and Caddy gateway from one source repository, plus managed Postgres and an object storage bucket. Only the gateway is public.

Does self-hosted Dagster have built-in authentication?

No. The open-source webserver ships no login, and anyone who reaches it can launch runs, read logs and terminate work. That is why it stays private here, behind a basic auth gateway. Set a strong DAGSTER_AUTH_PASSWORD before sharing the URL.

Why does the template include Postgres and object storage?

Postgres holds run history, the event log and schedule state, so nothing is lost when a container restarts. The bucket holds step stdout and the values assets pass between each other — each run worker is a separate process, so local disk serves neither purpose.

How is Dagster different from Airflow or Prefect?

All three are Apache 2.0 and self-hostable. Airflow orchestrates tasks and has the largest integration ecosystem; Prefect focuses on running arbitrary Python workflows. Dagster models the output — assets, their lineage, freshness and data-quality checks — which suits analytics engineering well.