🧙 A modern replacement for Airflow.

### Give your data team `magical` powers

Integrate and synchronize data from 3rd party sources

Build real-time and batch pipelines to transform data using Python, SQL, and R

Run, monitor, and orchestrate thousands of pipelines without losing sleep

1️⃣ 🏗️

Build

Have you met anyone who said they loved developing in Airflow?
That’s why we designed an easy developer experience that you’ll enjoy.

| | | | --- | --- | | Easy developer experience
Start developing locally with a single command or launch a dev environment in your cloud using Terraform.

Language of choice
Write code in Python, SQL, or R in the same data pipeline for ultimate flexibility.

Engineering best practices built-in
Each step in your pipeline is a standalone file containing modular code that’s reusable and testable with data validations. No more DAGs with spaghetti code. |

↓

2️⃣ 🔮

Preview

Stop wasting time waiting around for your DAGs to finish testing.
Get instant feedback from your code each time you run it.

| | | | --- | --- | | Interactive code
Immediately see results from your code’s output with an interactive notebook UI.

Data is a first-class citizen
Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.

Collaborate on cloud
Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment. |

↓

3️⃣ 🚀

Launch

Don’t have a large team dedicated to Airflow?
Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.

| | | | --- | --- | | Fast deploy
Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.

Scaling made simple
Transform very large datasets directly in your data warehouse or through a native integration with Spark.

Observability
Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI. |

|
# 🧙 Intro Mage is an open-source data pipeline tool for transforming and integrating data. 1. [Install](#%EF%B8%8F-install) 1. [Demo](#-demo) 1. [Tutorials](#-tutorials) 1. [Documentation](https://docs.mage.ai) 1. [Features](#-features) 1. [Core design principles](https://docs.mage.ai/design/core-design-principles) 1. [Core abstractions](https://docs.mage.ai/design/core-abstractions) 1. [Contributing](https://docs.mage.ai/community/contributing) Looking for help? The _fastest_ way to get started is by checking out our documentation [here](https://docs.mage.ai/getting-started/setup). Looking for quick examples? Open a [demo](https://demo.mage.ai/) project right in your browser or check out our [guides](https://docs.mage.ai/guides/overview). # 🎮 Demo ### Live demo Build and run a data pipeline with our [demo app](https://demo.mage.ai/). > WARNING > > The live demo is public to everyone, please don’t save anything sensitive (e.g. passwords, secrets, etc). ### Demo video (5 min) [![Mage quick start demo](https://github.com/mage-ai/assets/blob/main/overview/overview-video.png?raw=True)](https://youtu.be/GswOdShLGmg) _{Click the image to play video}
# 👩‍🏫 Tutorials - [Load data from API, transform it, and export it to PostgreSQL](https://docs.mage.ai/guides/load-api-data) - [Integrate Mage into an existing Airflow project](https://docs.mage.ai/integrations/airflow) - [Train model on Titanic dataset](https://docs.mage.ai/guides/train-model) - [Set up dbt models and orchestrate dbt runs](https://docs.mage.ai/integrations/dbt-models) Fire mage

# 🔮 [Features](https://docs.mage.ai/about/features) | | | | | --- | --- | --- | | 🎶 | [Orchestration](https://docs.mage.ai/design/data-pipeline-management) | Schedule and manage data pipelines with observability. | | 📓 | [Notebook](https://docs.mage.ai/about/features#notebook-for-building-data-pipelines) | Interactive Python, SQL, & R editor for coding data pipelines. | | 🏗️ | [Data integrations](https://docs.mage.ai/data-integrations/overview) | Synchronize data from 3rd party sources to your internal destinations. | | 🚰 | [Streaming pipelines](https://docs.mage.ai/guides/streaming-pipeline) | Ingest and transform real-time data. | | ❎ | [dbt](https://docs.mage.ai/dbt/overview) | Build, run, and manage your dbt models with Mage. | A sample data pipeline defined across 3 files ➝ 1. Load data ➝ ```python @data_loader def load_csv_from_file(): return pd.read_csv('default_repo/titanic.csv') ``` 1. Transform data ➝ ```python @transformer def select_columns_from_df(df, *args): return df[['Age', 'Fare', 'Survived']] ``` 1. Export data ➝ ```python @data_exporter def export_titanic_data_to_disk(df) -> None: df.to_csv('default_repo/titanic_transformed.csv') ``` What the data pipeline looks like in the UI ➝ data pipeline overview

New? We recommend reading about [blocks](https://docs.mage.ai/design/blocks) and learning from a [hands-on tutorial](https://docs.mage.ai/guides/load-api-data). [![Ask us questions on Slack](https://img.shields.io/badge/%20-Ask%20us%20questions%20on%20Slack-purple?style=for-the-badge&logo=slack&labelColor=6B50D7)](https://www.mage.ai/chat)
# 🏔️ [Core design principles](https://docs.mage.ai/design/core-design-principles) Every user experience and technical design decision adheres to these principles. | | | | | --- | --- | --- | | 💻 | [Easy developer experience](https://docs.mage.ai/design/core-design-principles#easy-developer-experience) | Open-source engine that comes with a custom notebook UI for building data pipelines. | | 🚢 | [Engineering best practices built-in](https://docs.mage.ai/design/core-design-principles#engineering-best-practices-built-in) | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. | | 💳 | [Data is a first-class citizen](https://docs.mage.ai/design/core-design-principles#data-is-a-first-class-citizen) | Designed from the ground up specifically for running data-intensive workflows. | | 🪐 | [Scaling is made simple](https://docs.mage.ai/design/core-design-principles#scaling-is-made-simple) | Analyze and process large data quickly for rapid iteration. |
# 🛸 [Core abstractions](https://docs.mage.ai/design/core-abstractions) These are the fundamental concepts that Mage uses to operate. | | | | --- | --- | | [Project](https://docs.mage.ai/design/core-abstractions#project) | Like a repository on GitHub; this is where you write all your code. | | [Pipeline](https://docs.mage.ai/design/core-abstractions#pipeline) | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. | | [Block](https://docs.mage.ai/design/core-abstractions#block) | A file with code that can be executed independently or within a pipeline. | | [Data product](https://docs.mage.ai/design/core-abstractions#data-product) | Every block produces data after it's been executed. These are called data products in Mage. | | [Trigger](https://docs.mage.ai/design/core-abstractions#trigger) | A set of instructions that determine when or how a pipeline should run. | | [Run](https://docs.mage.ai/design/core-abstractions#run) | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |
# 🤔 Frequently Asked Questions (FAQs) Check out our [FAQ page](https://docs.mage.ai/about/frequently-asked-questions) to find answers to some of our most asked questions.
# 🪪 License See the [LICENSE](LICENSE) file for licensing information. [ Water mage casting spell

](https://www.mage.ai/)