Deploy Prefect

Deploy and Host Prefect on Railway

Prefect is an open source workflow orchestration framework for Python. Decorate ordinary functions with @flow and @task, and Prefect handles scheduling, retries, caching, concurrency limits and observability — turning a script that "usually works" into a pipeline you can see, replay and trust. Teams use it for ETL and ELT jobs, ML training pipelines and any recurring Python work that has to run reliably.

Deploy Prefect on Railway and you get the full production topology, not a single container on SQLite. This template wires the API server, a dedicated background-services process and a process worker to managed Postgres and Redis, so the API stays stateless, the scheduler runs in its own container, and flow runs execute on a worker you scale independently. Self-host Prefect from the official prefecthq/prefect:3-latest image (github.com/PrefectHQ/prefect), with basic authentication on and a generated password so the UI is never open on first boot.

Getting Started with Prefect on Railway

Open the public URL of the prefect-server service. Prefect shows its own login field asking for an authentication string — paste the whole user:password value of PREFECT_SERVER_API_AUTH_STRING, not just the password. The dashboard is empty until something runs, so check Work Pools first: the worker creates default-process-pool on startup, and a green dot beside it means it is polling and the deployment is healthy end to end. Then point your own project at the server with PREFECT_API_URL set to https://your-app.up.railway.app/api and the same auth string, and run prefect deploy targeting that pool. Your deployment appears under Deployments, where the row menu offers Quick run. Watch it under Runs for a timeline of every task, live logs, parameters and artifacts.

About Hosting Prefect

Prefect separates orchestration from your code. Flows stay in your repository and run on your infrastructure; the server records state, schedules work and hands it to workers, so your data never leaves your account.

• Python-native API — no DSL, no YAML DAG files, just decorators
• Automatic retries, timeouts, caching and result persistence per task
• Cron, interval and RRule schedules plus event-driven automations
• Concurrency limits, run history, structured logs, timings and artifacts

prefect-server serves the REST API and UI on port 4200, started with --no-services so it stays stateless. prefect-services runs the scheduler, cancellation monitors, event triggers and automation actions. prefect-worker polls a work pool and launches each flow run as a subprocess. Postgres holds all orchestration state; Redis carries the event bus, concurrency leases and the docket lock that stops background services duplicating scheduled work.

Why Deploy Prefect on Railway

One click gives you the whole orchestration stack instead of a starting point.

• Managed Postgres and Redis, provisioned and connected for you
• Private networking between services, nothing extra exposed
• Managed TLS, a public URL and custom domains out of the box
• Basic auth enabled with a generated password on first deploy
• API, background services and worker scale independently

Common Use Cases

• Scheduled ETL and ELT jobs that pull from APIs, transform with pandas or Polars and load a warehouse
• dbt and data-quality runs triggered by an upstream ingestion flow through Prefect automations
• ML pipelines that retrain nightly and promote a model only when evaluation tasks pass
• Backfills where retries, caching and concurrency limits matter more than throughput

Dependencies for Prefect

• Prefect — prefecthq/prefect:3-latest, the official image, for the server, background services and worker: github.com/PrefectHQ/prefect
• PostgreSQL — 14.9+, storing flows, deployments, runs, logs, events and automations
• Redis — events broker and cache, causal ordering, concurrency leases, docket lock

Environment Variables Reference

Deployment Dependencies

• Image: prefecthq/prefect 3-latest, Python 3.12 base
• Source: github.com/PrefectHQ/prefect, Apache 2.0
• Docs: docs.prefect.io/v3/advanced/self-hosted
• Requires PostgreSQL 14.9+ and Redis 6+

Server Requirements to Self-Host Prefect

Sizing is driven by the worker, because your flow code runs inside it. Scale the worker before touching the control plane.

How to Self-Host Prefect Outside Railway

Everything ships in one image, so a minimal stack is three containers plus Postgres and Redis. Start the API server:

docker run -d --name prefect-server -p 4200:4200 \
  -e PREFECT_SERVER_API_HOST=0.0.0.0 \
  -e PREFECT_SERVER_API_AUTH_STRING="admin:choose-a-password" \
  -e PREFECT_SERVER_DATABASE_CONNECTION_URL="postgresql+asyncpg://prefect:pass@postgres:5432/prefect" \
  prefecthq/prefect:3-latest prefect server start --no-services

Then the background services and a worker against it:

docker run -d -e PREFECT_SERVER_DATABASE_CONNECTION_URL="postgresql+asyncpg://prefect:pass@postgres:5432/prefect" \
  -e PREFECT_SERVER_DOCKET_URL="redis://redis:6379/1" \
  prefecthq/prefect:3-latest prefect server services start

docker run -d -e PREFECT_API_URL="http://prefect-server:4200/api" \
  -e PREFECT_API_AUTH_STRING="admin:choose-a-password" \
  prefecthq/prefect:3-latest prefect worker start -p default-process-pool -t process

Is Prefect Free to Self-Host?

Prefect's engine and server are open source under Apache 2.0 and self-hosting is free with no feature gating — schedules, retries, automations, work pools and the full UI are all included. On Railway you pay only for the compute and storage the five services use. Prefect Cloud is the managed alternative, with a free Hobby tier and paid plans from around $100 per month that add SSO, audit logs and role-based access control.

Prefect vs Airflow vs Dagster

All three are Apache 2.0. Choose Prefect when your workflows are ordinary Python; Airflow has the bigger integration catalogue, Dagster is stronger when lineage is the point.

FAQ

What is Prefect and why self-host it? Prefect is an open source Python workflow orchestration tool that schedules, retries and monitors data pipelines. Self-hosting keeps your flow code, credentials and data inside your own infrastructure.

What does this Railway template deploy? A Prefect API and UI server, a background-services container running the scheduler and automations, a process worker, managed PostgreSQL and managed Redis, with basic auth already on.

Why does the template include Postgres and Redis? Postgres is required whenever Prefect runs as more than one process; SQLite cannot coordinate a separate API, services and worker. Redis carries the event bus, concurrency leases and the lock that stops a run being scheduled twice.

How do I run my own flows on this deployment? Set PREFECT_API_URL and PREFECT_API_AUTH_STRING in your project, then run prefect deploy targeting default-process-pool. The worker pulls your code at run time from the deployment's pull steps.

Can I add Python packages my flows depend on? Yes — set EXTRA_PIP_PACKAGES on the worker to a space-separated list and the image installs them at start. For heavier sets, build your own image from prefecthq/prefect:3-latest.

Is self-hosted Prefect production ready? Yes. The API server is stateless and runs at any replica count and the worker scales separately, but keep prefect-services at one replica unless the Redis docket URL is set.