Deploy DataHub

Deploy and Host DataHub on Railway

DataHub is the open-source metadata platform LinkedIn built to make a very large data estate searchable, now maintained by Acryl Data. It gives engineers, analysts and governance teams one place to find every table, topic, dashboard, ML feature and pipeline in a stack, with column-level lineage, ownership and access policies attached. Teams reach for it when "which table feeds this dashboard, and who owns it?" stops being answerable from memory.

Deploy DataHub on Railway and you get the full production topology, not a single-container demo. It runs nine services: the React frontend, the GMS metadata service, standalone MCE and MAE consumers, the actions runner that executes ingestion, a system-update service that migrates the schema and builds search indices, plus Kafka in KRaft mode, OpenSearch 2 and managed MySQL. Writes enter GMS, travel through Kafka to the MCE consumer, land in MySQL, and are re-published as change logs the MAE consumer indexes into OpenSearch — the event-driven path DataHub runs at LinkedIn scale.

Getting Started with DataHub on Railway

Set DATAHUB_ADMIN_PASSWORD on the frontend before deploying — the container refuses to start without it, because the upstream image ships a datahub / datahub login no configuration file otherwise overrides. Open the frontend's URL and sign in as datahub. The home page starts empty: DataHub discovers nothing on its own.

Your first useful action is to catalog something. Open Data Sources, click Create Source, and pick a connector — Snowflake, BigQuery, dbt, Postgres, Looker, Airflow and about seventy others ship built in. Enter the connection details, save and press run; the actions service builds a per-run virtual environment, runs the connector and streams progress into the run history. When the status turns Success, the assets appear in search within seconds. To see the product working before wiring up a real warehouse, create a source of type demo-data: it loads a sample catalog of Hive tables, Airflow DAGs and Looker dashboards with lineage attached. Opening one of those datasets to inspect its columns, owners and lineage exercises every service in the template.

About Hosting DataHub

DataHub is a metadata platform, not just a catalog UI. Every fact about an asset is modelled as an aspect on an entity URN and delivered as an event, which suits teams who want to build on the metadata rather than only browse it. Self-hosting makes sense when the catalog must reach systems inside a VPC, when the metadata is sensitive, or when internal tooling needs the GraphQL and OpenAPI surfaces.

Key capabilities:

• Search across datasets, dashboards, charts, pipelines and ML models
• Column-level lineage and impact analysis before schema changes
• Ownership, tags, glossary terms and domains
• Ingestion from ~70 sources, from the UI or the CLI
• Access policies, tokens and optional OIDC sign-on
• GraphQL API, OpenAPI endpoints and a Python emitter

The split: datahub-frontend serves the app and login, datahub-gms owns reads, writes and the GraphQL API, datahub-mce-consumer applies change proposals to MySQL, datahub-mae-consumer indexes change logs into OpenSearch, datahub-actions runs ingestion jobs, datahub-system-update applies migrations.

Why Deploy DataHub on Railway

Railway removes the operational tax of self-hosting DataHub.

• Nine services, Kafka, OpenSearch and MySQL provisioned in one click
• Private networking; only the UI and API are public
• Persistent volumes for Kafka and OpenSearch, managed backups for MySQL
• Health checks on every HTTP service
• Scale each tier independently
• Usage-based pricing, no per-seat licence

Common Use Cases for Self-Hosted DataHub

• Warehouse discovery — index Snowflake, BigQuery, Redshift or Databricks so analysts find the right table without asking in Slack
• Impact analysis — trace lineage from a source column through dbt models to the dashboards that would break if it were dropped
• Governance — tag PII columns and enforce ownership before an audit asks
• Platform engineering — push custom metadata via the Python emitter, read it back over GraphQL

Dependencies for DataHub

• acryldata/datahub-frontend-react:v1.7.0 — web UI, extended to replace the default password
• acryldata/datahub-gms:v1.7.0 — metadata service and GraphQL API
• acryldata/datahub-mce-consumer:v1.7.0 — applies change proposals
• acryldata/datahub-mae-consumer:v1.7.0 — indexes change logs and lineage
• acryldata/datahub-actions:v1.7.0-slim — ingestion executor
• acryldata/datahub-upgrade:v1.7.0 — migrations and index builds
• confluentinc/cp-kafka:8.2.2 — event log, KRaft, no ZooKeeper
• opensearchproject/opensearch:2.19.3 — search, browse, graph index
• Railway managed MySQL 9.4 — durable aspect store

All DataHub images are pinned to one release: the components exchange Avro records and refuse to run as a mixed set, so bump them together.

Environment Variables Reference

Deployment Dependencies

• Source: github.com/datahub-project/datahub
• Docs: docs.datahub.com
• Images: hub.docker.com/u/acryldata
• CLI: pip install "acryl-datahub[<source>]"

Hardware Requirements for Self-Hosting DataHub

GMS and OpenSearch are the memory-hungry pair. Both size their heap from the container limit, so raising a service's memory raises its heap.

Self-Hosting DataHub

Upstream's quickstart runs the same components locally. The following is a shell session:

python3 -m pip install --upgrade "acryl-datahub[datahub-rest]"
datahub docker quickstart

Against any running instance, ingestion is a YAML recipe plus one command. Create a token under Settings → Access Tokens, then save this as recipe.yml:

source:
  type: postgres
  config:
    host_port: db.example.com:5432
    database: analytics
    username: readonly
    password: ${POSTGRES_PASSWORD}

sink:
  type: datahub-rest
  config:
    server: https://your-gms-domain.up.railway.app
    token: ${DATAHUB_TOKEN}

Run datahub ingest -c recipe.yml from a laptop or CI. The Data Sources screen stores the same recipe body.

How Much Does DataHub Cost to Self-Host?

DataHub is free and open source under Apache 2.0 — no seat limits, feature gating or licence key. Acryl Data sells a managed cloud version with extra observability, but the self-hosted build is the complete catalog, lineage and governance product. On Railway you pay only for the compute, memory and volume storage the nine services use.

FAQ

What is DataHub?

An open-source metadata platform: a searchable catalog of the datasets, dashboards, pipelines and ML models in a data stack, with lineage, ownership and governance policies attached.

Why does DataHub need Kafka, OpenSearch and MySQL?

MySQL stores every metadata aspect, Kafka carries each change as an event, and OpenSearch backs search, browse and the lineage graph. There is no embedded-database mode.

How do I ingest metadata into self-hosted DataHub?

From the Data Sources screen, which runs the connector inside the actions service on a schedule, or with datahub ingest -c recipe.yml from the CLI using an access token.

How do I secure a self-hosted DataHub instance?

Set DATAHUB_ADMIN_PASSWORD before the first deploy so the shipped default is never live, keep METADATA_SERVICE_AUTH_ENABLED on, and move logins to your identity provider with the AUTH_OIDC_* variables once several people need access.

Can I scale DataHub on Railway?

Yes. The MCE and MAE consumers run as their own services, so ingestion and indexing scale independently of the UI and API. Raise replicas or memory on the tier that is the bottleneck.