Deploy Prefect — Self-Hosted Python Workflow Orchestration

Deploy and Host Prefect on Railway

Prefect is a modern open-source workflow orchestration platform for Python — a Pythonic alternative to Airflow where flows are just Python functions decorated with @flow. Build, schedule, and monitor data pipelines and automations in code you write in your own repo, with a clean UI for runs, logs, retries, and scheduling. This template deploys the Prefect server (UI, API, scheduler) with PostgreSQL and basic authentication configured — a private, password-protected orchestration backend, ready for your workers to connect, in minutes.

What This Template Deploys

Both connect over Railway's private network. The Prefect server orchestrates and displays your flows, backed by PostgreSQL for persistence, and is protected by basic auth. Your workers — the processes that run your flow code — connect to this server from wherever you run them.

About Hosting

Prefect's self-hosted architecture has two things to understand — authentication and the server/worker split — and this template sets both up right.

Prefect's open-source server has no auth by default — this template adds it. This is critical: the Prefect OSS server ships with no authentication, no RBAC, and no API keys (those are Prefect Cloud features), so a naive deploy leaves your orchestration API and UI open to anyone with the URL. This template sets PREFECT_SERVER_API_AUTH_STRING to a username and password, locking the server behind basic auth from the first deploy. A common gotcha when connecting workers: the client reads PREFECT_API_AUTH_STRING (not the SERVER variant), and if PREFECT_API_KEY is set anywhere in a worker's environment it takes precedence and causes 401 errors — so leave it unset for self-hosted.

This deploys the server — workers run separately. The template runs the Prefect server (UI, API, scheduler) — the orchestration brain, not where your flows execute. Your flow code runs on workers, which you run as separate processes (another Railway service, your own machine, or any compute) that poll this server outbound — no inbound access needed. Point a worker at your server with PREFECT_API_URL set to your Railway domain's /api, and it picks up scheduled runs. This keeps the server light and lets you size workers per workload.

PostgreSQL with the right driver prefix. Prefect stores flow runs, deployments, schedules, logs, and state in PostgreSQL via PREFECT_API_DATABASE_CONNECTION_URL, which must use the postgresql+asyncpg:// driver prefix — a plain postgresql:// URL won't connect. This template wires it correctly to the managed Postgres, replacing the default SQLite that's unsuitable for a persistent PaaS deployment.

The UI needs its API URL, and state persists. PREFECT_UI_API_URL (and PREFECT_SERVER_API_HOST=0.0.0.0) are set so the web UI reaches the API through your Railway domain — without this, the UI loads but can't reach the backend. All orchestration state — flow runs, deployments, schedules, and logs — lives in PostgreSQL, so it survives redeploys and is the single backup target; the server itself is stateless.

Typical cost: ~$10–15/month on Railway for the server and PostgreSQL — the server is light, and workers are sized separately to your workloads. Prefect's open-source server is Apache-2.0 and free, versus Prefect Cloud's paid plans.

How It Compares

Airflow is powerful but heavy and DAG-centric. Kestra is a strong orchestrator but YAML-first, authored in its UI. Prefect Cloud is the managed version with RBAC but paid. Prefect self-hosted's edge is Pythonic orchestration — flows are ordinary Python functions in your repo, with retries, scheduling, and observability — lightweight and free, with your workflows on infrastructure you own. Choose Prefect if your team writes orchestration in Python; choose a YAML orchestrator if you prefer declarative config.

Deploy in Under 5 Minutes

1. Click Deploy on Railway — the Prefect server and PostgreSQL build and wire together (~4 minutes)
2. Confirm PREFECT_SERVER_API_AUTH_STRING is set and the Postgres URL uses postgresql+asyncpg://
3. Open your Railway URL on port 4200 and log in with your basic-auth credentials
4. Set up a worker (a separate service or machine) with PREFECT_API_URL pointing at your server's /api
5. Deploy a Python @flow, schedule it, and watch runs in the UI

Keep PREFECT_API_KEY unset on self-hosted workers, or they'll 401 against your server.

Common Use Cases

• Python data pipelines — orchestrate ETL and data workflows written as Python functions
• Scheduled automation — run and monitor recurring Python jobs with retries and alerting
• Airflow replacement — Pythonic orchestration without Airflow's operational weight

Configuration

The server is open by default — this locks it down. PREFECT_SERVER_API_AUTH_STRING adds basic auth (OSS Prefect has no built-in auth otherwise). On workers, use PREFECT_API_AUTH_STRING and leave PREFECT_API_KEY unset, or they'll 401.

Postgres needs the asyncpg driver, and workers run separately. The connection URL must use postgresql+asyncpg://. This template deploys the server; run workers as separate apps pointed at it via PREFECT_API_URL.

Dependencies for Prefect Hosting

• Railway account — ~$10–15/month for the server and PostgreSQL; workers sized separately
• PostgreSQL (included and wired with the asyncpg driver)
• One or more workers (separate service or machine) to execute flow code
• Optional: Redis/Valkey messaging for scaling the server

Deployment Dependencies

• Prefect GitHub Repository
• Prefect Self-Hosting Guide
• Prefect Server Authentication
• Railway Private Networking

Implementation Details

The template runs the official prefecthq/prefect:3-latest image with prefect server start on port 4200, backed by a Railway-managed PostgreSQL database. Persistence uses PREFECT_API_DATABASE_CONNECTION_URL with the required postgresql+asyncpg:// driver prefix; a plain postgresql:// URL fails to connect, and the default SQLite backend is unsuitable for a persistent PaaS deployment. PREFECT_SERVER_API_HOST=0.0.0.0 and PREFECT_UI_API_URL (the Railway domain's /api) are set so the API is reachable and the UI can talk to it.

The open-source Prefect server has no built-in authentication, RBAC, or API keys — those are Cloud features — so PREFECT_SERVER_API_AUTH_STRING is set to enable basic auth. When connecting workers, the client reads PREFECT_API_AUTH_STRING rather than the SERVER variant, and a present PREFECT_API_KEY (even empty) takes precedence and causes 401s, so it should be left unset. Architecturally, this deploys the server (UI, API, scheduler); workers that execute flow code run as separate processes polling the server outbound via PREFECT_API_URL, needing no inbound access, so they scale per workload. All orchestration state persists in PostgreSQL, the single backup target; Redis/Valkey can be added as a messaging broker when scaling.

Frequently Asked Questions

Where do my flows actually run? On workers, not on this server. The server orchestrates and displays runs; you run workers as separate processes (another service or machine) that poll the server via PREFECT_API_URL and execute your Python flow code.

Why do my workers get 401 errors? Two common causes on self-hosted: the client reads PREFECT_API_AUTH_STRING (not the SERVER variant), and any PREFECT_API_KEY set in the worker environment takes precedence and causes 401s. Set the auth string and leave the API key unset.

How is Prefect different from Kestra or Airflow? Prefect flows are ordinary Python functions in your repo, lighter than Airflow's DAGs. Kestra is declarative YAML authored in its UI. Choose Prefect if you'd rather write orchestration in Python.

Why Deploy Prefect on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Prefect on Railway you get a Pythonic orchestration server with the hard parts solved — PostgreSQL wired with the right driver, basic auth enabled, and the UI and API configured. Run your Python flows on workers that connect to a private, password-protected server, self-hosted on infrastructure you own.