Deploy and Host Firecrawl on Railway

Firecrawl is an open-source web scraping and crawling API that turns any website into clean, LLM-ready Markdown or structured JSON — a self-hosted alternative to Apify and Scrapy. Point it at a URL and it renders JavaScript, follows links, strips clutter, and returns content ready for a RAG pipeline or AI agent. This template deploys the full five-service Firecrawl stack, correctly wired, so the multi-service setup that trips up most self-hosters just works.

What This Template Deploys

All five run on Railway's private network. Only the API is public; the rest stay internal. This is the current Firecrawl architecture — wiring five services together is what makes self-hosting it hard, and what this template solves.

About Hosting

Firecrawl is powerful, but self-hosting it means running a coordinated five-service stack — and two specifics decide whether it holds up under real load.

It's a five-service architecture, and the pieces are specific. The API orchestrates jobs, Playwright renders pages, NUQ Postgres holds the job queue, Redis handles caching and rate limits, and RabbitMQ brokers async crawl messages. The Postgres piece matters: Firecrawl uses ghcr.io/firecrawl/nuq-postgres with a pg_cron schema, not a stock Postgres image. This template wires all five correctly, which is the entire difficulty of self-hosting Firecrawl.

Playwright is the memory hog — plan for 8 GB+ under load. Headless Chromium is by far the most memory-intensive component. 4 GB may hold for light use, but under sustained crawl load it's insufficient and the Playwright service gets OOM-killed mid-crawl. Budget 8 GB or more for production crawling, and scale the Playwright service's memory first.

The Supabase auth warning is normal, not a bug. Self-hosted Firecrawl runs with USE_DB_AUTHENTICATION=false and logs a warning about missing Supabase auth. This is expected on self-hosted instances — scraping works fully without it. Don't chase that warning; it's not an error.

Optional AI extraction needs an LLM key. Basic scrape and crawl need no AI. The LLM-powered "extract" format (pull structured data by schema) only activates when you set OPENAI_API_KEY (or point OPENAI_BASE_URL at a compatible endpoint, including local Ollama). Leave it unset for pure scraping.

Typical cost: ~$15–25/month on Railway across the five services, more if you scale Playwright's memory for heavy crawling. Firecrawl is open source; Apify and similar SaaS crawlers bill per usage.

How It Compares

Apify and ScrapingBee are managed but bill per use and keep your crawls on their infrastructure. Scrapy is powerful but a Python framework you build and maintain yourself. Firecrawl's edge is turning pages into clean, LLM-ready Markdown out of the box with a simple API — self-hosted here at flat cost, with your scraped data staying on your own infrastructure.

Deploy in Under 5 Minutes

1. Click Deploy on Railway — all five services build and wire together automatically (~5 minutes)
2. Confirm the API, Playwright, NUQ Postgres, Redis, and RabbitMQ services are healthy
3. For heavy crawling, raise the Playwright service's memory to 8 GB+
4. Optionally set OPENAI_API_KEY to enable AI-powered structured extraction
5. POST a URL to the /v1/scrape or /v1/crawl endpoint and get back clean Markdown or JSON

The Supabase auth warning in the logs is normal on self-hosted — scraping works without it.

Common Use Cases

• RAG data ingestion — scrape docs and sites into clean Markdown to chunk, embed, and feed a knowledge base
• AI agent web access — give an agent a scraping API that returns LLM-ready content instead of raw HTML
• Content aggregation — collect articles, blog posts, or listings from many sites into structured data
• Automation pipelines — connect Firecrawl to n8n or your backend to trigger crawls and route the output
• Private crawling — keep scraped data on your own infrastructure instead of a third-party crawl service

Configuration

Give Playwright enough memory. Headless Chromium is the memory bottleneck — 4 GB is insufficient under sustained crawl load. Budget 8 GB+ for production and scale the Playwright service first, or crawls get OOM-killed mid-job.

The Supabase auth warning is expected. Self-hosted Firecrawl runs with USE_DB_AUTHENTICATION=false and logs a missing-auth warning — normal, and scraping works fully. It's not an error to fix.

Dependencies for Firecrawl Hosting

• Railway account — ~$15–25/month across five services, more for heavy crawling
• Adequate memory for Playwright (8 GB+ recommended for production crawls)
• The full stack — API, Playwright, NUQ Postgres, Redis, RabbitMQ (all included)
• Optional: an OpenAI-compatible API key for AI-powered structured extraction

Deployment Dependencies

• Firecrawl GitHub Repository
• Firecrawl Self-Hosting Guide
• Firecrawl API Documentation
• Railway Private Networking

Implementation Details

The template runs Firecrawl's current five-service architecture: the API (ghcr.io/firecrawl/firecrawl, port 3002) orchestrating jobs, the Playwright service rendering JavaScript, NUQ Postgres (ghcr.io/firecrawl/nuq-postgres) holding the job queue with a pg_cron schema, Redis for caching and rate limiting, and RabbitMQ brokering async crawl messages. Only the API is exposed publicly; the other four communicate over Railway's private network via reference variables (REDIS_URL, PLAYWRIGHT_MICROSERVICE_URL, NUQ_DATABASE_URL).

The NUQ Postgres image is specific — it carries the pg_cron schema Firecrawl's queue depends on, so stock Postgres won't work. Playwright is the resource-critical component: headless Chromium's memory scales with concurrent rendering, which is why production crawling needs 8 GB or more. USE_DB_AUTHENTICATION=false is the self-hosted default and produces an expected Supabase warning, not an error. AI extraction is opt-in: setting OPENAI_API_KEY (or OPENAI_BASE_URL for a compatible or local model) enables schema-based extraction, while plain scrape and crawl run without any AI config.

Frequently Asked Questions

Why is it five services? Firecrawl separates concerns: the API orchestrates, Playwright renders JavaScript, NUQ Postgres queues jobs, Redis caches and rate-limits, and RabbitMQ brokers async work. Wiring all five correctly is the hard part of self-hosting — this template does it for you.

Why does it need so much memory? Headless Chromium (Playwright) is memory-hungry, especially under concurrent crawls. 4 GB is fine for light use but insufficient for sustained crawling — budget 8 GB+ and scale the Playwright service first.

There's a Supabase auth warning — is it broken? No. Self-hosted Firecrawl runs with USE_DB_AUTHENTICATION=false, which logs that warning by design. Scraping works fully without it; it's expected, not an error.

Do I need an OpenAI key? Only for the AI extract format that pulls structured data by schema. Plain scrape and crawl to Markdown need no AI key — set OPENAI_API_KEY only if you want extraction.

What can I do with the output? Firecrawl returns clean Markdown or JSON ideal for RAG pipelines, embeddings, and AI agents — no HTML parsing needed. Pair it with a vector store to build a knowledge base from any set of sites.

How does it compare to Apify? Apify is managed and billed per usage; Firecrawl self-hosted is flat-cost and keeps data on your infrastructure, with output already formatted for LLMs.

Why Deploy Firecrawl on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Firecrawl on Railway you get the full five-service scraping stack wired correctly — API, Playwright, NUQ Postgres, Redis, and RabbitMQ over a private network, with automatic HTTPS. Turn any website into clean, LLM-ready Markdown through a simple API, self-hosted on infrastructure you own.