Deploy Apache Airflow

Airflow 3.4 with Celery: API server, scheduler, worker, Postgres and Redis

Deploy Apache Airflow

Just deployed

/var/lib/postgresql/data

Just deployed

/opt/airflow/logs

Just deployed

/opt/airflow/logs

Just deployed

/data

Just deployed

Deploy and Host Apache Airflow on Railway

Apache Airflow is an open-source platform for writing, scheduling and monitoring workflows as Python code. You describe a pipeline as a DAG of tasks. Airflow runs it on a schedule or on demand, retries failed tasks, and shows every run, task and log in a web UI. It's widely used for ETL and data engineering.

About Hosting Apache Airflow

A production Airflow setup has several moving parts. This template deploys Airflow 3.4.0 with the CeleryExecutor, following Airflow's official Docker Compose setup. The API server serves the UI and REST API. The scheduler queues tasks, the DAG processor parses DAG files, the triggerer runs deferred tasks, and a Celery worker runs the tasks themselves. Postgres stores Airflow's metadata and Redis is the Celery broker. Every service talks over Railway's private network. Secrets and the admin password are generated, and the database is migrated on the first start. Task logs are kept on volumes, so they survive redeploys.

Common Use Cases

  • Scheduling ETL and ELT pipelines between databases, warehouses and APIs
  • Orchestrating dbt, Spark or machine-learning jobs with retries and alerts
  • Replacing cron jobs with versioned, observable Python workflows

Dependencies for Apache Airflow Hosting

  • PostgreSQL (Railway Postgres): Airflow's metadata database and Celery result backend
  • Redis (Railway Redis): the Celery broker that hands tasks to the worker

Deployment Dependencies

Implementation Details

Services in this template:

  • Airflow: the public service. It runs airflow api-server (web UI, REST API and the Execution API that workers call), runs airflow db migrate and creates the admin user before it starts.
  • Airflow Scheduler: runs airflow scheduler.
  • Airflow DAG Processor: runs airflow dag-processor.
  • Airflow Triggerer: runs airflow triggerer, with its logs on a volume.
  • Airflow Worker: runs airflow celery worker (4 task slots), with task logs on a volume. The UI reads them from the worker over the private network.
  • Postgres and Redis: Railway's databases, each on its own volume.

After deploy:

  1. Open the Airflow service's public URL and sign in as admin. The password is _AIRFLOW_WWW_USER_PASSWORD in the Airflow service's Variables tab.
  2. Unpause and trigger the example DAG railway_hello to check that tasks run.
  3. To run your own DAGs, set DAGS_GIT_URL (an HTTPS Git URL), optionally DAGS_GIT_BRANCH (default main) and DAGS_GIT_SUBDIR, on the Airflow service, then redeploy all Airflow services. Airflow pulls the repository every 60 seconds.
  4. For more task capacity, raise the Airflow Worker's replicas or set AIRFLOW__CELERY__WORKER_CONCURRENCY on it.

Keep AIRFLOW_FERNET_SECRET unchanged: it encrypts the connections and variables stored in Postgres. Every Airflow service builds from the template's GitHub repo, so when a new Airflow version is merged there, Railway shows you an update. Read the repo's CHANGELOG first, because the Airflow service migrates the database on its first start.

Why Deploy Apache Airflow on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Apache Airflow on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.


Template Content

More templates in this category

View Template
N8N Main + Worker
Deploy and Host N8N with Inactive worker.

jakemerson
121
View Template
Evolution API with n8n
Automate WhatsApp workflows with Evolution API, n8n, and Postgres.

codestorm
113
View Template
Postgres Backup
Cron-based PostgreSQL backup to bucket storage

Railway Templates
867