Deploy Firecrawl

Deploy and Host Firecrawl on Railway

Firecrawl is an open-source web data API that turns any URL into clean, LLM-ready Markdown or structured JSON. It handles the parts of web scraping that usually eat an afternoon — JavaScript rendering, sitemap discovery, following links to a depth limit, stripping navigation and ads, respecting robots.txt, and returning page metadata with the content. Teams building RAG systems, AI agents and competitor monitoring use it as the ingestion layer in front of a vector database; it is the self-hostable alternative to Apify, Bright Data and Diffbot.

Self-host Firecrawl on Railway with the production topology already wired together: the REST API, five worker roles pulling jobs off a Postgres-backed queue, a headless Chromium renderer, RabbitMQ for job-completion notifications, and Redis for rate limiting. Because a self-hosted Firecrawl accepts any bearer token, this template does not expose the API directly — a Caddy gateway holds the only public domain, checks every request against an API key you control, and forwards it over the private network. Deploy Firecrawl and you get a working https:// endpoint with an API key, not a checklist of hardening tasks.

Getting Started with Firecrawl on Railway

There is no web UI — Firecrawl is an API, ready the moment the gateway service turns green. Open that service in Railway and copy two things: its public domain and its FIRECRAWL_API_KEY. Then ask it to scrape a page:

curl -X POST https://your-gateway-domain.up.railway.app/v2/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

A healthy deployment answers "success": true with the page as Markdown; repeat the call without the header and it must return 401. POST /v2/map returns every URL Firecrawl can discover on a domain, and POST /v2/crawl queues a multi-page job, returning an id you poll at GET /v2/crawl/ until status reads completed. The official SDKs work unchanged against your host:

from firecrawl import Firecrawl

app = Firecrawl(api_key="YOUR_KEY", api_url="https://your-gateway-domain.up.railway.app")
docs = app.crawl("https://docs.example.com", limit=50)

About Hosting Firecrawl

Firecrawl closes the gap between "a web page exists" and "a language model can read it". Point it at a URL and it fetches the page, renders JavaScript when needed, converts the DOM to Markdown and returns metadata such as HTTP status, title and canonical URL. Point it at a domain and it walks the sitemap and links, returning the same output per page. Teams self-host it when volume makes a metered SaaS expensive, when content is too sensitive for a third party, or to keep the crawler beside the vector database it feeds.

Key capabilities:

• Scrape a URL to Markdown, HTML, links, screenshots or structured JSON
• Crawl a site with depth, path and page-count limits, honouring robots.txt
• Map a domain to its discoverable URLs in one fast call
• Batch scrape many URLs in one job, with webhooks when it finishes
• Extract schema-defined fields; PDFs and DOCX parse automatically

The template splits Firecrawl's processes into separate services rather than one container. api answers HTTP and enqueues work; nuq-worker performs the scrapes; worker orchestrates crawls; nuq-prefetch-worker feeds jobs to workers and nuq-reconciler-worker returns abandoned ones to the queue; extract-worker handles schema extraction; playwright-service runs Chromium. nuq-postgres is the queue itself — Firecrawl's own PostgreSQL image with pg_cron and the queue schema baked in, so a plain Postgres cannot replace it. RabbitMQ carries completion notifications and Redis backs rate limiting.

Why Deploy Firecrawl on Railway

Railway removes the operational work a self-hosted crawler usually adds.

• Eleven pre-wired services with private networking configured
• Persistent volumes for the queue database and broker survive redeploys
• Health checks everywhere, so a stuck worker restarts instead of idling
• Scaling per role — add scraping capacity without touching the API
• One public HTTPS domain with a managed certificate; backends stay private

Common Use Cases

• RAG ingestion: crawl product docs on a schedule and push clean Markdown into a vector database
• AI agents with live web access: give an agent a /v2/scrape tool so it reads pages at request time, not from stale training data
• Competitor and price monitoring: map a catalogue, scrape each page, diff the output on a cron
• Content migration: convert a legacy site or wiki into Markdown for a static site generator

Dependencies for Firecrawl

• ghcr.io/firecrawl/firecrawl:latest — API and every worker role
• ghcr.io/firecrawl/playwright-service:latest — headless Chromium renderer
• ghcr.io/firecrawl/nuq-postgres:latest — PostgreSQL 17 with pg_cron and the queue schema
• rabbitmq:3-management — job completion notifications
• Railway managed Redis — rate limiting and caching
• caddy:2-alpine from the gateway source repository — API key enforcement

Environment Variables Reference

Deployment Dependencies

• Source: github.com/firecrawl/firecrawl
• Self-hosting guide: docs.firecrawl.dev/contributing/self-host

Hardware Requirements for Self-Hosting Firecrawl

Chromium is the expensive part; queue and API are modest.

Scale per role: raise replicas on nuq-worker and give playwright-service more memory when crawls back up. Leave nuq-prefetch-worker and nuq-reconciler-worker at one replica — both are singletons.

Self-Hosting Firecrawl with Docker

Upstream ships a Compose file that runs every process in one container — the quickest local try:

git clone https://github.com/firecrawl/firecrawl.git
cd firecrawl
cp apps/api/.env.example .env
docker compose up -d
curl -X POST http://localhost:3002/v2/scrape -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

That local stack is unauthenticated and keeps no volumes, so it suits evaluation and nothing public. Splitting those processes into separate services — each with its own health check, resources and replica count — plus the API key gateway, is what this template does for you.

How Much Does Firecrawl Cost to Self-Host?

Firecrawl is open source under the AGPL-3.0 licence, so there are no seat or page-credit fees. The hosted cloud starts free with a small credit allowance and moves to paid tiers as volume grows; self-hosting replaces that with infrastructure cost only. Two hosted conveniences — managed proxy rotation and the Fire-engine renderer — are not part of the open-source build.

FAQ

What is Firecrawl? An open-source API that scrapes and crawls websites, returning clean Markdown or structured JSON built for language models.

What does this Railway template deploy? Eleven services: the Firecrawl API, five worker roles, a Playwright renderer, a queue database, RabbitMQ, Redis, and a Caddy gateway holding the public domain.

How do I authenticate requests to a self-hosted Firecrawl? Self-hosted Firecrawl accepts any bearer token, so the gateway checks the Authorization: Bearer header against FIRECRAWL_API_KEY. Rotate the key by changing that variable; the gateway refuses to start if it is empty.

Why does the template include Postgres, RabbitMQ and Redis? Postgres is the job queue Firecrawl schedules scrapes through, and needs the extensions in Firecrawl's own image. RabbitMQ notifies the API when a job finishes. Redis backs rate limiting.

Does self-hosted Firecrawl respect robots.txt? Yes, unless a request opts out with ignoreRobotsTxt. You remain responsible for the terms of the sites you crawl.

How do I enable LLM extraction on self-hosted Firecrawl? Set OPENAI_API_KEY on api and extract-worker — optionally with OPENAI_BASE_URL and MODEL_NAME for a compatible provider.