--- title: "Deploy DataHub" description: "DataHub: Searchable catalog of your tables, dashboards & pipelines" category: "Analytics" url: https://railway.com/deploy/datahub --- # Deploy DataHub DataHub: Searchable catalog of your tables, dashboards & pipelines **[Deploy DataHub on Railway](https://railway.com/template/datahub)** - **Creator:** A3A - **Category:** Analytics ## Template content ### datahub-mce-consumer https://raw.githubusercontent.com/datahub-project/datahub/v1.7.0/datahub-web-react/public/assets/logos/datahublogo.svg - **Image:** acryldata/datahub-mce-consumer:v1.7.0 - **Health check:** /actuator/health ### datahub-mae-consumer https://raw.githubusercontent.com/datahub-project/datahub/v1.7.0/datahub-web-react/public/assets/logos/datahublogo.svg - **Image:** acryldata/datahub-mae-consumer:v1.7.0 - **Health check:** /actuator/health ### datahub-gms https://raw.githubusercontent.com/datahub-project/datahub/v1.7.0/datahub-web-react/public/assets/logos/datahublogo.svg - **Image:** acryldata/datahub-gms:v1.7.0 - **Health check:** /health - **Public domain:** Yes ### datahub-system-update https://raw.githubusercontent.com/datahub-project/datahub/v1.7.0/datahub-web-react/public/assets/logos/datahublogo.svg - **Source:** https://github.com/gridalpha/datahub-railway ### kafka https://cdn.jsdelivr.net/gh/homarr-labs/dashboard-icons/svg/kafka.svg - **Image:** confluentinc/cp-kafka:8.2.2 - **Start command:** `/bin/sh -c 'mkdir -p /var/lib/kafka/data/logs && exec /etc/confluent/docker/run'` ### MySQL https://cdn.jsdelivr.net/gh/homarr-labs/dashboard-icons/svg/mysql.svg - **Image:** mysql:9.4 - **Start command:** `docker-entrypoint.sh mysqld --innodb-use-native-aio=0 --disable-log-bin --performance_schema=0 --innodb-buffer-pool-size=1G` ### datahub-frontend https://raw.githubusercontent.com/datahub-project/datahub/v1.7.0/datahub-web-react/public/assets/logos/datahublogo.svg - **Source:** https://github.com/gridalpha/datahub-railway - **Health check:** /admin - **Public domain:** Yes ### opensearch https://cdn.jsdelivr.net/gh/homarr-labs/dashboard-icons/svg/opensearch.svg - **Source:** https://github.com/gridalpha/datahub-railway - **Health check:** / ### datahub-actions https://raw.githubusercontent.com/datahub-project/datahub/v1.7.0/datahub-web-react/public/assets/logos/datahublogo.svg - **Image:** acryldata/datahub-actions:v1.7.0-slim ## Documentation ![DataHub logo](https://repository-images.githubusercontent.com/46398090/6eab4c80-5005-11ea-9953-a9b699abe985) # Deploy and Host DataHub on Railway DataHub is the open-source metadata platform LinkedIn built to make a very large data estate searchable, now maintained by Acryl Data. It gives engineers, analysts and governance teams one place to find every table, topic, dashboard, ML feature and pipeline in a stack, with column-level lineage, ownership and access policies attached. Teams reach for it when "which table feeds this dashboard, and who owns it?" stops being answerable from memory. Deploy DataHub on Railway and you get the full production topology, not a single-container demo. It runs nine services: the React frontend, the GMS metadata service, standalone MCE and MAE consumers, the actions runner that executes ingestion, a system-update service that migrates the schema and builds search indices, plus Kafka in KRaft mode, OpenSearch 2 and managed MySQL. Writes enter GMS, travel through Kafka to the MCE consumer, land in MySQL, and are re-published as change logs the MAE consumer indexes into OpenSearch — the event-driven path DataHub runs at LinkedIn scale. ![Railway topology of nine DataHub services with Kafka, OpenSearch and MySQL](https://res.cloudinary.com/rroe4rtk/image/upload/v1788245139/datahub-architecture.png) ## Getting Started with DataHub on Railway Set `DATAHUB_ADMIN_PASSWORD` on the frontend before deploying — the container refuses to start without it, because the upstream image ships a `datahub` / `datahub` login no configuration file otherwise overrides. Open the frontend's URL and sign in as `datahub`. The home page starts empty: DataHub discovers nothing on its own. Your first useful action is to catalog something. Open **Data Sources**, click **Create Source**, and pick a connector — Snowflake, BigQuery, dbt, Postgres, Looker, Airflow and about seventy others ship built in. Enter the connection details, save and press run; the actions service builds a per-run virtual environment, runs the connector and streams progress into the run history. When the status turns **Success**, the assets appear in search within seconds. To see the product working before wiring up a real warehouse, create a source of type `demo-data`: it loads a sample catalog of Hive tables, Airflow DAGs and Looker dashboards with lineage attached. Opening one of those datasets to inspect its columns, owners and lineage exercises every service in the template. ![DataHub home page listing owned assets and ingested platforms](https://res.cloudinary.com/rroe4rtk/image/upload/v1788245140/datahub-home.png) ![DataHub search results across sixty catalogued data assets](https://res.cloudinary.com/rroe4rtk/image/upload/v1788245141/datahub-search.png) ![Hive dataset profile showing columns, owners and lineage](https://res.cloudinary.com/rroe4rtk/image/upload/v1788245142/datahub-dataset-profile.png) ![DataHub data sources page with a successful ingestion run](https://res.cloudinary.com/rroe4rtk/image/upload/v1788245143/datahub-data-sources.png) ## About Hosting DataHub DataHub is a metadata *platform*, not just a catalog UI. Every fact about an asset is modelled as an aspect on an entity URN and delivered as an event, which suits teams who want to build on the metadata rather than only browse it. Self-hosting makes sense when the catalog must reach systems inside a VPC, when the metadata is sensitive, or when internal tooling needs the GraphQL and OpenAPI surfaces. Key capabilities: - Search across datasets, dashboards, charts, pipelines and ML models - Column-level lineage and impact analysis before schema changes - Ownership, tags, glossary terms and domains - Ingestion from ~70 sources, from the UI or the CLI - Access policies, tokens and optional OIDC sign-on - GraphQL API, OpenAPI endpoints and a Python emitter The split: **datahub-frontend** serves the app and login, **datahub-gms** owns reads, writes and the GraphQL API, **datahub-mce-consumer** applies change proposals to MySQL, **datahub-mae-consumer** indexes change logs into OpenSearch, **datahub-actions** runs ingestion jobs, **datahub-system-update** applies migrations. ## Why Deploy DataHub on Railway Railway removes the operational tax of self-hosting DataHub. - Nine services, Kafka, OpenSearch and MySQL provisioned in one click - Private networking; only the UI and API are public - Persistent volumes for Kafka and OpenSearch, managed backups for MySQL - Health checks on every HTTP service - Scale each tier independently - Usage-based pricing, no per-seat licence ## Common Use Cases for Self-Hosted DataHub - **Warehouse discovery** — index Snowflake, BigQuery, Redshift or Databricks so analysts find the right table without asking in Slack - **Impact analysis** — trace lineage from a source column through dbt models to the dashboards that would break if it were dropped - **Governance** — tag PII columns and enforce ownership before an audit asks - **Platform engineering** — push custom metadata via the Python emitter, read it back over GraphQL ## Dependencies for DataHub - `acryldata/datahub-frontend-react:v1.7.0` — web UI, extended to replace the default password - `acryldata/datahub-gms:v1.7.0` — metadata service and GraphQL API - `acryldata/datahub-mce-consumer:v1.7.0` — applies change proposals - `acryldata/datahub-mae-consumer:v1.7.0` — indexes change logs and lineage - `acryldata/datahub-actions:v1.7.0-slim` — ingestion executor - `acryldata/datahub-upgrade:v1.7.0` — migrations and index builds - `confluentinc/cp-kafka:8.2.2` — event log, KRaft, no ZooKeeper - `opensearchproject/opensearch:2.19.3` — search, browse, graph index - Railway managed MySQL 9.4 — durable aspect store All DataHub images are pinned to one release: the components exchange Avro records and refuse to run as a mixed set, so bump them together. ### Environment Variables Reference | Variable | Service | Purpose | |---|---|---| | `DATAHUB_ADMIN_PASSWORD` | frontend | Password for the built-in `datahub` account. Required | | `DATAHUB_ADMIN_USER` | frontend | Username; leave as `datahub` | | `DATAHUB_SYSTEM_CLIENT_SECRET` | gms, frontend, actions | Secret the services authenticate to GMS with | | `DATAHUB_TOKEN_SERVICE_SIGNING_KEY` | gms, mce | Signs access tokens; changing it invalidates issued ones | | `METADATA_SERVICE_AUTH_ENABLED` | gms, frontend | Rejects unauthenticated API calls. Keep `true` | | `AUTH_OIDC_ENABLED` | frontend | Set with the `AUTH_OIDC_*` family to move login to SSO | | `DATAHUB_OBJECT_STORAGE_URI` | gms | Point at `s3://` to keep uploaded assets | ### Deployment Dependencies - Source: [github.com/datahub-project/datahub](https://github.com/datahub-project/datahub) - Docs: [docs.datahub.com](https://docs.datahub.com) - Images: [hub.docker.com/u/acryldata](https://hub.docker.com/u/acryldata) - CLI: `pip install "acryl-datahub[]"` ## Hardware Requirements for Self-Hosting DataHub | Resource | Minimum | Recommended | |---|---|---| | CPU | 4 vCPU total | 8 vCPU total | | RAM | 8 GB total | 16 GB total | | Storage | 10 GB across the Kafka and OpenSearch volumes | 50 GB+ for a big catalog | | Runtime | JVM 17+, Kafka 3.7+, OpenSearch 2.x, MySQL 8+ | — | GMS and OpenSearch are the memory-hungry pair. Both size their heap from the container limit, so raising a service's memory raises its heap. ## Self-Hosting DataHub Upstream's quickstart runs the same components locally. The following is a shell session: ``` python3 -m pip install --upgrade "acryl-datahub[datahub-rest]" datahub docker quickstart ``` Against any running instance, ingestion is a YAML recipe plus one command. Create a token under **Settings → Access Tokens**, then save this as `recipe.yml`: ``` source: type: postgres config: host_port: db.example.com:5432 database: analytics username: readonly password: ${POSTGRES_PASSWORD} sink: type: datahub-rest config: server: https://your-gms-domain.up.railway.app token: ${DATAHUB_TOKEN} ``` Run `datahub ingest -c recipe.yml` from a laptop or CI. The Data Sources screen stores the same recipe body. ## How Much Does DataHub Cost to Self-Host? DataHub is free and open source under Apache 2.0 — no seat limits, feature gating or licence key. Acryl Data sells a managed cloud version with extra observability, but the self-hosted build is the complete catalog, lineage and governance product. On Railway you pay only for the compute, memory and volume storage the nine services use. ## FAQ **What is DataHub?** An open-source metadata platform: a searchable catalog of the datasets, dashboards, pipelines and ML models in a data stack, with lineage, ownership and governance policies attached. **Why does DataHub need Kafka, OpenSearch and MySQL?** MySQL stores every metadata aspect, Kafka carries each change as an event, and OpenSearch backs search, browse and the lineage graph. There is no embedded-database mode. **How do I ingest metadata into self-hosted DataHub?** From the Data Sources screen, which runs the connector inside the actions service on a schedule, or with `datahub ingest -c recipe.yml` from the CLI using an access token. **How do I secure a self-hosted DataHub instance?** Set `DATAHUB_ADMIN_PASSWORD` before the first deploy so the shipped default is never live, keep `METADATA_SERVICE_AUTH_ENABLED` on, and move logins to your identity provider with the `AUTH_OIDC_*` variables once several people need access. **Can I scale DataHub on Railway?** Yes. The MCE and MAE consumers run as their own services, so ingestion and indexing scale independently of the UI and API. Raise replicas or memory on the tier that is the bottleneck. ## Similar templates - [Typesense vs Meilisearch](https://railway.com/deploy/typesense-vs-meilisearch) — self-hosted Typesense vs Meilisearch - [Betterlytics](https://railway.com/deploy/betterlytics) — Betterlytics is a cookieless analytics platform GDPR-compliant. - [Finance Tracker](https://railway.com/deploy/finance-tracker-1) — Private multi-user household finance ledger with budgets and CSV import. Open this page in a browser: https://railway.com/deploy/datahub