--- title: "Deploy Firecrawl — Web Scraper & Crawler API" description: "Self-host Firecrawl — scrape & crawl any site to clean Markdown" category: "AI/ML" url: https://railway.com/deploy/firecrawl-scraper-api --- # Deploy Firecrawl — Web Scraper & Crawler API Self-host Firecrawl — scrape & crawl any site to clean Markdown **[Deploy Firecrawl — Web Scraper & Crawler API on Railway](https://railway.com/template/firecrawl-scraper-api)** - **Creator:** SilverBanana - **Category:** AI/ML ## Template content ### Redis https://cdn.sanity.io/images/sy1jschh/production/0ce0bfdcfbdbf69662b1116671f97c2dd788b655-157x157.svg - **Image:** redis:8.2.1 - **Start command:** `/bin/sh -c "rm -rf $RAILWAY_VOLUME_MOUNT_PATH/lost+found/ && exec docker-entrypoint.sh redis-server --requirepass $REDIS_PASSWORD --save 60 1 --dir $RAILWAY_VOLUME_MOUNT_PATH"` ### RabbitMQ https://cdn.iconscout.com/icon/free/png-256/free-rabbitmq-icon-svg-download-png-282296.png - **Image:** rabbitmq:3-alpine ### NUQ-Postgres https://devicons.railway.app/PostgreSQL?variant=light - **Image:** ghcr.io/firecrawl/nuq-postgres ### Firecrawl Playwright https://repository-images.githubusercontent.com/403222057/5927bf3d-8b0d-46d8-9777-bc9c23531fd6 - **Image:** ghcr.io/firecrawl/playwright-service - **Public domain:** Yes ### Firecrawl API https://res.cloudinary.com/dh2nt6hgh/image/upload/v1758463059/New_Project_2_bxehtv.webp - **Image:** ghcr.io/firecrawl/firecrawl - **Public domain:** Yes ## Documentation # Deploy and Host Firecrawl on Railway Firecrawl is an open-source web scraping and crawling API that turns any website into clean, LLM-ready Markdown or structured JSON — a self-hosted alternative to Apify and Scrapy. Point it at a URL and it renders JavaScript, follows links, strips clutter, and returns content ready for a RAG pipeline or AI agent. This template deploys the full five-service Firecrawl stack, correctly wired, so the multi-service setup that trips up most self-hosters just works. --- ## What This Template Deploys | Service | Purpose | | --- | --- | | **Firecrawl API** | Orchestrates scrape and crawl jobs, serves the REST API on port `3002` | | **Playwright Service** | Headless Chromium for rendering JavaScript-heavy pages | | **NUQ Postgres** | The job queue with pg_cron scheduling (a specific image, not plain Postgres) | | **Redis** | Rate limiting and caching | | **RabbitMQ** | Brokers async crawl jobs between workers | All five run on Railway's private network. Only the API is public; the rest stay internal. This is the current Firecrawl architecture — wiring five services together is what makes self-hosting it hard, and what this template solves. --- ## About Hosting Firecrawl is powerful, but self-hosting it means running a coordinated five-service stack — and two specifics decide whether it holds up under real load. **It's a five-service architecture, and the pieces are specific.** The API orchestrates jobs, Playwright renders pages, NUQ Postgres holds the job queue, Redis handles caching and rate limits, and RabbitMQ brokers async crawl messages. The Postgres piece matters: Firecrawl uses `ghcr.io/firecrawl/nuq-postgres` with a pg_cron schema, not a stock Postgres image. This template wires all five correctly, which is the entire difficulty of self-hosting Firecrawl. **Playwright is the memory hog — plan for 8 GB+ under load.** Headless Chromium is by far the most memory-intensive component. 4 GB may hold for light use, but under sustained crawl load it's insufficient and the Playwright service gets OOM-killed mid-crawl. Budget 8 GB or more for production crawling, and scale the Playwright service's memory first. **The Supabase auth warning is normal, not a bug.** Self-hosted Firecrawl runs with `USE_DB_AUTHENTICATION=false` and logs a warning about missing Supabase auth. This is expected on self-hosted instances — scraping works fully without it. Don't chase that warning; it's not an error. **Optional AI extraction needs an LLM key.** Basic scrape and crawl need no AI. The LLM-powered "extract" format (pull structured data by schema) only activates when you set `OPENAI_API_KEY` (or point `OPENAI_BASE_URL` at a compatible endpoint, including local Ollama). Leave it unset for pure scraping. Typical cost: **~$15–25/month** on Railway across the five services, more if you scale Playwright's memory for heavy crawling. Firecrawl is open source; Apify and similar SaaS crawlers bill per usage. --- ## How It Compares | | Firecrawl (self-hosted) | Apify | Scrapy | ScrapingBee | | --- | --- | --- | --- | --- | | Cost model | Flat infra | Per usage | Free (self-run) | Per request | | Output | Clean Markdown / JSON | Varies | Raw, you parse | HTML | | JS rendering | Yes (Playwright) | Yes | Add-on | Yes | | LLM-ready extraction | Yes | Add-on | No | No | | Setup | This template, one click | Managed | Heavy coding | Managed | | Self-hostable | Yes | Limited | Yes | No | Apify and ScrapingBee are managed but bill per use and keep your crawls on their infrastructure. Scrapy is powerful but a Python framework you build and maintain yourself. Firecrawl's edge is turning pages into clean, LLM-ready Markdown out of the box with a simple API — self-hosted here at flat cost, with your scraped data staying on your own infrastructure. --- ## Deploy in Under 5 Minutes 1. Click **Deploy on Railway** — all five services build and wire together automatically (~5 minutes) 2. Confirm the API, Playwright, NUQ Postgres, Redis, and RabbitMQ services are healthy 3. For heavy crawling, raise the Playwright service's memory to 8 GB+ 4. Optionally set `OPENAI_API_KEY` to enable AI-powered structured extraction 5. POST a URL to the `/v1/scrape` or `/v1/crawl` endpoint and get back clean Markdown or JSON The Supabase auth warning in the logs is normal on self-hosted — scraping works without it. --- ## Common Use Cases - **RAG data ingestion** — scrape docs and sites into clean Markdown to chunk, embed, and feed a knowledge base - **AI agent web access** — give an agent a scraping API that returns LLM-ready content instead of raw HTML - **Content aggregation** — collect articles, blog posts, or listings from many sites into structured data - **Automation pipelines** — connect Firecrawl to n8n or your backend to trigger crawls and route the output - **Private crawling** — keep scraped data on your own infrastructure instead of a third-party crawl service --- ## Configuration | Variable | Required | Description | | --- | --- | --- | | `REDIS_URL` | Auto-injected | Redis connection via Railway reference variable | | `REDIS_RATE_LIMIT_URL` | Auto-injected | Redis for rate limiting | | `PLAYWRIGHT_MICROSERVICE_URL` | Auto-injected | Internal URL of the Playwright service | | `NUQ_DATABASE_URL` | Auto-injected | NUQ Postgres connection for the job queue | | `USE_DB_AUTHENTICATION` | Pre-set | `false` — expected for self-hosted; the auth warning is normal | | `OPENAI_API_KEY` | Optional | Enables the AI-powered `extract` format; leave unset for plain scraping | | `OPENAI_BASE_URL` | Optional | Point at a compatible endpoint (e.g. Ollama) for extraction | > **Give Playwright enough memory.** Headless Chromium is the memory bottleneck — 4 GB is insufficient under sustained crawl load. Budget 8 GB+ for production and scale the Playwright service first, or crawls get OOM-killed mid-job. > **The Supabase auth warning is expected.** Self-hosted Firecrawl runs with `USE_DB_AUTHENTICATION=false` and logs a missing-auth warning — normal, and scraping works fully. It's not an error to fix. --- ## Dependencies for Firecrawl Hosting - Railway account — ~$15–25/month across five services, more for heavy crawling - Adequate memory for Playwright (8 GB+ recommended for production crawls) - The full stack — API, Playwright, NUQ Postgres, Redis, RabbitMQ (all included) - Optional: an OpenAI-compatible API key for AI-powered structured extraction ### Deployment Dependencies - [Firecrawl GitHub Repository](https://github.com/firecrawl/firecrawl) - [Firecrawl Self-Hosting Guide](https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md) - [Firecrawl API Documentation](https://docs.firecrawl.dev/) - [Railway Private Networking](https://docs.railway.com/guides/private-networking) ### Implementation Details The template runs Firecrawl's current five-service architecture: the API (`ghcr.io/firecrawl/firecrawl`, port `3002`) orchestrating jobs, the Playwright service rendering JavaScript, NUQ Postgres (`ghcr.io/firecrawl/nuq-postgres`) holding the job queue with a pg_cron schema, Redis for caching and rate limiting, and RabbitMQ brokering async crawl messages. Only the API is exposed publicly; the other four communicate over Railway's private network via reference variables (`REDIS_URL`, `PLAYWRIGHT_MICROSERVICE_URL`, `NUQ_DATABASE_URL`). The NUQ Postgres image is specific — it carries the pg_cron schema Firecrawl's queue depends on, so stock Postgres won't work. Playwright is the resource-critical component: headless Chromium's memory scales with concurrent rendering, which is why production crawling needs 8 GB or more. `USE_DB_AUTHENTICATION=false` is the self-hosted default and produces an expected Supabase warning, not an error. AI extraction is opt-in: setting `OPENAI_API_KEY` (or `OPENAI_BASE_URL` for a compatible or local model) enables schema-based extraction, while plain scrape and crawl run without any AI config. --- ## Frequently Asked Questions **Why is it five services?** Firecrawl separates concerns: the API orchestrates, Playwright renders JavaScript, NUQ Postgres queues jobs, Redis caches and rate-limits, and RabbitMQ brokers async work. Wiring all five correctly is the hard part of self-hosting — this template does it for you. **Why does it need so much memory?** Headless Chromium (Playwright) is memory-hungry, especially under concurrent crawls. 4 GB is fine for light use but insufficient for sustained crawling — budget 8 GB+ and scale the Playwright service first. **There's a Supabase auth warning — is it broken?** No. Self-hosted Firecrawl runs with `USE_DB_AUTHENTICATION=false`, which logs that warning by design. Scraping works fully without it; it's expected, not an error. **Do I need an OpenAI key?** Only for the AI `extract` format that pulls structured data by schema. Plain scrape and crawl to Markdown need no AI key — set `OPENAI_API_KEY` only if you want extraction. **What can I do with the output?** Firecrawl returns clean Markdown or JSON ideal for RAG pipelines, embeddings, and AI agents — no HTML parsing needed. Pair it with a vector store to build a knowledge base from any set of sites. **How does it compare to Apify?** Apify is managed and billed per usage; Firecrawl self-hosted is flat-cost and keeps data on your infrastructure, with output already formatted for LLMs. --- ## Why Deploy Firecrawl on Railway? Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it. By deploying Firecrawl on Railway you get the full five-service scraping stack wired correctly — API, Playwright, NUQ Postgres, Redis, and RabbitMQ over a private network, with automatic HTTPS. Turn any website into clean, LLM-ready Markdown through a simple API, self-hosted on infrastructure you own. ## Similar templates - [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform. - [stella](https://railway.com/deploy/stella) — Self-host stella with web, API, Postgres, Redis, and object storage. - [Hermes Agent | OpenClaw Alternative with Dashboard](https://railway.com/deploy/hermes-agent-or-openclaw-alternative-wit) — Self-Hosted Hermes AI Agent for Telegram, Discord & Slack Open this page in a browser: https://railway.com/deploy/firecrawl-scraper-api