Deploy Apache Airflow

Deploy and Host Apache Airflow on Railway

Apache Airflow is the workflow orchestrator most data teams standardise on. You write pipelines as Python — a DAG of tasks with dependencies, retries and schedules — and Airflow runs them in order, on time, across as many machines as you give it. Because DAGs are ordinary Python files they live in version control and can call any library you already use. Self-host Airflow and you get the scheduler, REST API and React UI with no per-task pricing and no data leaving your infrastructure.

Deploy Airflow on Railway and you get the production CeleryExecutor topology, not a single container that quietly drops work. Five services run from one image: an API server serving the UI, REST API and task-execution endpoint; a scheduler; a DAG processor; a triggerer; and a Celery worker. Managed PostgreSQL holds the metadata, managed Redis is the broker, and an object storage bucket keeps task logs so any service can read a log any worker wrote. Only the API server is public.

Getting Started with Apache Airflow on Railway

Open the API server's public URL and you land on the Airflow sign-in page. The first administrator is created on first boot from AIRFLOW_ADMIN_USERNAME and AIRFLOW_ADMIN_PASSWORD on the airflow-apiserver service — read that password from the service's Variables tab, sign in, and change it from the user menu. The home page's Health panel should show MetaDatabase, Scheduler, Triggerer and Dag Processor green within a minute or two, confirming every service found the database and the broker. Click Dags to find three example pipelines, paused by default. Toggle railway_sales_etl on and press play: it builds a synthetic day of orders, rolls them up by region and prints the totals. Open the run, click transform and then Logs — that log comes from the bucket, proving the worker, broker and remote logging are wired correctly. railway_parallel_fanout then shows task mapping across the worker tier.

About Hosting Apache Airflow

Airflow separates what runs from where it runs: DAGs describe dependencies and schedules, the scheduler decides which task instances are ready, and the executor hands them to workers. Running all of that in one process is fine on a laptop and a liability in production, because one crash takes scheduling and execution down together. This template splits the roles as Airflow documents them.

• Pipelines as Python code — dynamic DAGs, TaskFlow decorators, task mapping, typed XComs
• Real scheduling — cron, data intervals, timetables, catchup, backfills, dependency-aware retries
• Deferrable tasks — long waits move to the triggerer, so waiting an hour costs no worker slot
• Hundreds of providers — AWS, GCP, Azure, Snowflake, dbt, Kubernetes, HTTP and SQL
• REST API, CLI and role-based access — Admin, Op, User, Viewer and Public roles

Each supporting service has a job. PostgreSQL stores DAG definitions, run history, task state, connections and variables, and is also the Celery result backend. Redis is the broker the scheduler pushes queued tasks onto and workers pull from. The bucket holds task logs, which matters because a Railway volume attaches to exactly one service. The DAG processor parses Python in its own process so a broken file cannot stall scheduling; the triggerer runs the loop deferrable operators wait on.

Why Deploy Apache Airflow on Railway

Railway removes the infrastructure work self-hosting Airflow usually implies.

• Managed PostgreSQL, Redis and object storage provisioned with the template
• Five services, private networking and TLS with no YAML to write
• Push to the source repository and every service rebuilds together
• Scale the worker on its own when pipelines get heavier
• No Kubernetes cluster, Helm chart or certificates

Common Use Cases for Self-Hosted Apache Airflow

• ELT and warehouse loading — pull from APIs and databases on a schedule, then run dbt models in order
• Machine learning pipelines — scheduled retraining, feature generation and batch scoring, with retries
• Report and dashboard refresh — recompute aggregates overnight so BI tools open instantly
• Operational automation — reconciliation, data-quality checks and backups that need audit history

Dependencies for Apache Airflow

• apache/airflow:3.3.1, extended by a small source repository, gridalpha/airflow-railway, which adds the boot-time configuration Railway needs and bakes in the example DAGs
• PostgreSQL 18 (Railway managed) — metadata database and Celery result backend
• Redis 8 (Railway managed) — Celery broker
• Railway object storage bucket — remote task logs

Environment Variables Reference

Any AIRFLOW__SECTION__KEY variable you set overrides what the image would otherwise compute, so the whole Airflow configuration reference stays available.

Deployment Dependencies

• Apache Airflow on GitHub
• Official documentation
• apache/airflow on Docker Hub
• Provider packages

Hardware Requirements for Self-Hosting Apache Airflow

Airflow's own guidance is at least 4 GB of RAM and 2 CPUs for the cluster as a whole; the worker is the service to grow.

Memory is dominated by DAG parsing and by what your tasks do, so heavy work belongs in a service the task calls, not the worker itself.

Self-Hosting Apache Airflow with Docker

The Airflow project publishes a Compose file for the same topology. These shell commands start it locally:

curl -LfO https://airflow.apache.org/docs/apache-airflow/3.3.1/docker-compose.yaml
mkdir -p ./dags ./logs ./plugins ./config
echo "AIRFLOW_UID=$(id -u)" > .env
docker compose up -d

Adding Python dependencies means extending the image rather than installing at container start:

FROM apache/airflow:3.3.1
COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt
COPY dags/ /opt/airflow/dags/

On Railway, fork the source repository, add your packages and DAGs, and push: every service rebuilds from the same commit, keeping the five roles on matching versions.

How Much Does Apache Airflow Cost to Self-Host?

Airflow is free and open source under the Apache 2.0 licence, with no paid tier, seat limits or task quotas. Managed alternatives such as Astronomer, Cloud Composer and Amazon MWAA start around $300 a month for a small environment. Self-hosting on Railway costs only the compute, database and storage used.

FAQ

What is Apache Airflow? An open-source platform for authoring, scheduling and monitoring workflows. Pipelines are Python code — directed acyclic graphs of tasks with dependencies — and Airflow schedules them, retries failures and records the history in a web UI.

Why does the template include PostgreSQL, Redis and object storage? PostgreSQL is the metadata database Airflow requires beyond local testing, Redis is the broker moving queued tasks to workers, and the bucket holds task logs so the API server can show logs written on another service.

How do I add my own DAGs to self-hosted Apache Airflow? Fork the source repository, drop your Python files into dags/ and push. Or set AIRFLOW_DAGS_GIT_REPO_URL on all five Airflow services to add your git repository as a second, versioned DAG bundle without rebuilding.

How do I scale Apache Airflow workers on Railway? Raise AIRFLOW__CELERY__WORKER_CONCURRENCY to run more tasks per container, or increase the worker's replica count. Keep the scheduler, triggerer and DAG processor at one replica each unless configured for high availability.