Deploy Apache Airflow
Airflow 3.4 with Celery: API server, scheduler, worker, Postgres and Redis
Airflow DAG Processor
Just deployed
Just deployed
/var/lib/postgresql/data
Airflow Worker
Just deployed
/opt/airflow/logs
Airflow Triggerer
Just deployed
/opt/airflow/logs
Just deployed
Redis
Just deployed
/data
Airflow Scheduler
Just deployed
Deploy and Host Apache Airflow on Railway
Apache Airflow is an open-source platform for writing, scheduling and monitoring workflows as Python code. You describe a pipeline as a DAG of tasks. Airflow runs it on a schedule or on demand, retries failed tasks, and shows every run, task and log in a web UI. It's widely used for ETL and data engineering.
About Hosting Apache Airflow
A production Airflow setup has several moving parts. This template deploys Airflow 3.4.0 with the CeleryExecutor, following Airflow's official Docker Compose setup. The API server serves the UI and REST API. The scheduler queues tasks, the DAG processor parses DAG files, the triggerer runs deferred tasks, and a Celery worker runs the tasks themselves. Postgres stores Airflow's metadata and Redis is the Celery broker. Every service talks over Railway's private network. Secrets and the admin password are generated, and the database is migrated on the first start. Task logs are kept on volumes, so they survive redeploys.
Common Use Cases
- Scheduling ETL and ELT pipelines between databases, warehouses and APIs
- Orchestrating dbt, Spark or machine-learning jobs with retries and alerts
- Replacing cron jobs with versioned, observable Python workflows
Dependencies for Apache Airflow Hosting
- PostgreSQL (Railway Postgres): Airflow's metadata database and Celery result backend
- Redis (Railway Redis): the Celery broker that hands tasks to the worker
Deployment Dependencies
- Airflow image: apache/airflow 3.4.0 (official), pinned by digest in aalfath/apache-airflow-railway-template
- Airflow docs: Running Airflow in Docker and the 3.4.0 release notes
Implementation Details
Services in this template:
- Airflow: the public service. It runs
airflow api-server(web UI, REST API and the Execution API that workers call), runsairflow db migrateand creates the admin user before it starts. - Airflow Scheduler: runs
airflow scheduler. - Airflow DAG Processor: runs
airflow dag-processor. - Airflow Triggerer: runs
airflow triggerer, with its logs on a volume. - Airflow Worker: runs
airflow celery worker(4 task slots), with task logs on a volume. The UI reads them from the worker over the private network. - Postgres and Redis: Railway's databases, each on its own volume.
After deploy:
- Open the Airflow service's public URL and sign in as
admin. The password is_AIRFLOW_WWW_USER_PASSWORDin the Airflow service's Variables tab. - Unpause and trigger the example DAG
railway_helloto check that tasks run. - To run your own DAGs, set
DAGS_GIT_URL(an HTTPS Git URL), optionallyDAGS_GIT_BRANCH(defaultmain) andDAGS_GIT_SUBDIR, on the Airflow service, then redeploy all Airflow services. Airflow pulls the repository every 60 seconds. - For more task capacity, raise the Airflow Worker's replicas or set
AIRFLOW__CELERY__WORKER_CONCURRENCYon it.
Keep AIRFLOW_FERNET_SECRET unchanged: it encrypts the connections and variables stored in Postgres. Every Airflow service builds from the template's GitHub repo, so when a new Airflow version is merged there, Railway shows you an update. Read the repo's CHANGELOG first, because the Airflow service migrates the database on its first start.
Why Deploy Apache Airflow on Railway?
Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.
By deploying Apache Airflow on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.
Template Content
Airflow DAG Processor
aalfath/apache-airflow-railway-templateAirflow Worker
aalfath/apache-airflow-railway-templateAirflow Triggerer
aalfath/apache-airflow-railway-templateRedis
redis:8.2Airflow Scheduler
aalfath/apache-airflow-railway-template