--- title: "Deploy Crawl4AI" description: "Open-source LLM-friendly web crawler with REST API, playground and MCP." category: "AI/ML" url: https://railway.com/deploy/crawl4ai-1 --- # Deploy Crawl4AI Open-source LLM-friendly web crawler with REST API, playground and MCP. **[Deploy Crawl4AI on Railway](https://railway.com/template/crawl4ai-1)** Machine-readable deploy manifest (JSON, validated by TemplateCI): https://railway.com/deploy/crawl4ai-1/manifest.json - **Creator:** Salman Al Fariz's Projects - **Category:** AI/ML ## Template content ### Crawl4AI https://raw.githubusercontent.com/salmanalfariz24/crawl4ai-railway/main/assets/crawl4ai-icon.png - **Source:** salmanalfariz24/crawl4ai-railway - **Health check:** /health - **Public domain:** Yes ## Documentation # Deploy and Host Crawl4AI with Railway Crawl4AI is an open-source web crawler and scraper that turns web pages into clean, LLM-ready Markdown and structured data. Its self-hosted server exposes a REST API, a browser playground, and an MCP endpoint, so AI agents, RAG pipelines, and automation tools can crawl the web on demand. ## About Hosting Crawl4AI Hosting Crawl4AI means running the official Crawl4AI Docker image, which bundles a FastAPI server, headless Chromium, and a private Redis instance managed by supervisord. Since version 0.9 the server refuses to accept outside connections unless an API token is configured, so every request except the health check must send `Authorization: Bearer `. Chromium needs generous memory and shared memory. This template pins the image by version and digest, generates the API token for you, gives the container 1 GiB of `/dev/shm`, gates deploys on the `/health` endpoint, and exposes the API on a public HTTPS domain. No database or volume is needed. ## Common Use Cases - Turn web pages into clean Markdown for RAG ingestion and LLM context. - Give AI agents (Claude Code and other MCP clients) a web-crawling tool over MCP. - Extract structured data from pages with natural-language prompts through the `/llm/job` endpoint (requires an LLM API key). - Run crawls from automation tools such as n8n over plain HTTP. - Capture screenshots and PDFs of pages from a server-side headless browser. ## Dependencies for Crawl4AI Hosting - The official Crawl4AI Docker image (`unclecode/crawl4ai`), pinned by version and digest in this repository's `Dockerfile`. - Bundled inside the image: headless Chromium (Playwright), Redis (loopback only, password-protected), Gunicorn with Uvicorn workers, and supervisord. - Optional: an OpenAI or Anthropic API key for LLM-based extraction and filtering. ### Deployment Dependencies - Crawl4AI repository: https://github.com/unclecode/crawl4ai - Crawl4AI documentation: https://docs.crawl4ai.com/ - Crawl4AI self-hosting guide: https://docs.crawl4ai.com/core/self-hosting/ - Template source and changelog: https://github.com/salmanalfariz24/crawl4ai-railway ### Implementation Details **Plan requirement.** Crawl4AI recommends at least 4 GB of RAM for the container. Deploy on the Railway Hobby plan or higher; the Free and Trial plans allow 0.5 GB and 1 GB of RAM per service. **Scaling.** Keep one replica. Background jobs (`/crawl/job`, `/llm/job`) are stored in the container's own Redis, and Railway does not route a client back to the same replica. Railway gives the single replica more CPU and memory as needed, up to your plan's limit. **Variables.** | Variable | Set by template | Purpose | | ------------------------ | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `CRAWL4AI_API_TOKEN` | Generated 64-character hex secret | Bearer token for every endpoint except `/health`. Without it the server listens only inside the container and the deploy fails its healthcheck. | | `SECRET_KEY` | Generated 64-character hex secret | Signs the short-lived internal tokens Crawl4AI's MCP tools use to call its own API. Upstream asks for a fixed value in real deployments. | | `PORT` | `11235` | Tells Railway which port to route traffic and healthchecks to. Crawl4AI always listens on 11235. | | `RAILWAY_SHM_SIZE_BYTES` | `1073741824` | Gives the container 1 GiB of `/dev/shm` for Chromium, matching Crawl4AI's `--shm-size=1g` recommendation. | | `OPENAI_API_KEY` | Empty (optional) | Enables LLM features with the default provider `openai/gpt-4o-mini`. | | `ANTHROPIC_API_KEY` | Empty (optional) | Enables LLM features with Anthropic models. Also set `LLM_PROVIDER`. | | `LLM_PROVIDER` | Empty (optional) | Default LLM in LiteLLM `provider/model` format. Requests can only select models from this provider's family. Leave empty to use `openai/gpt-4o-mini`. | **Find your token.** Railway dashboard → your Crawl4AI service → **Variables** → `CRAWL4AI_API_TOKEN`. **Call the API.** Replace `` with the domain shown under **Settings → Networking**. ```bash export CRAWL4AI_URL="https://" export CRAWL4AI_API_TOKEN="" # Health (no token) curl "$CRAWL4AI_URL/health" # Crawl one or more URLs (max 100 per request) curl -X POST "$CRAWL4AI_URL/crawl" \ -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"urls": ["https://example.com"]}' # Markdown for a single URL curl -X POST "$CRAWL4AI_URL/md" \ -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` **Connect an MCP client.** The MCP endpoint is `https:///mcp/sse` (SSE transport). It requires the same `Authorization: Bearer` header. For Claude Code: ```bash claude mcp add --transport sse crawl4ai "https:///mcp/sse" \ --header "Authorization: Bearer " ``` Railway closes HTTP requests after 15 minutes, so long MCP sessions over SSE are cut periodically; Claude Code reconnects automatically. **Web UI.** `https:///playground` (request builder) and `https:///dashboard` (monitoring) load without a token; their API calls need the token. **Private networking.** Other services in the same Railway project can call `http://${{Crawl4AI.RAILWAY_PRIVATE_DOMAIN}}:11235` with the same Bearer header. ## Why Deploy Crawl4AI on Railway? Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it. By deploying Crawl4AI on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway. This template uses Crawl4AI (https://github.com/unclecode/crawl4ai). This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai). ## Similar templates - [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform. - [stella](https://railway.com/deploy/stella) — Self-host stella with web, API, Postgres, Redis, and object storage. - [Hermes Agent | OpenClaw Alternative with Dashboard](https://railway.com/deploy/hermes-agent-or-openclaw-alternative-wit) — Self-Hosted Hermes AI Agent for Telegram, Discord & Slack Open this page in a browser: https://railway.com/deploy/crawl4ai-1