Deploy and Host Crawl4AI on Railway

Crawl4AI is the most-starred open-source web crawler built for LLMs. It renders pages with a real headless Chromium, strips boilerplate and returns clean Markdown, structured JSON (CSS/XPath or LLM-based extraction), screenshots and PDFs — everything a RAG pipeline or AI agent needs to read the web. This template deploys the official Crawl4AI REST API server so any language or agent framework can crawl through a simple HTTP call.

About Hosting Crawl4AI

The template runs unclecode/crawl4ai as a single service on port 11235 with a public domain and a /health check. A random CRAWL4AI_API_TOKEN is generated for you and required as a Bearer token on every request, so your crawler is not open to the internet. Optional provider keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_TOKEN, GROQ_API_KEY) enable LLM extraction strategies. Chromium is memory hungry: budget 1–2 GB RAM for comfortable concurrent crawls. No database or volume is needed — Crawl4AI is stateless, and Railway's usage-based pricing means you only pay while it runs.

Common Use Cases

Feed clean Markdown from any URL into RAG / vector databases
Give AI agents (LangChain, CrewAI, OpenAI Agents, MCP tools) a "read this page" tool
Scrape product, pricing or documentation pages into structured JSON
Generate screenshots and PDFs of pages on demand

Dependencies for Crawl4AI Hosting

Docker image unclecode/crawl4ai:latest (bundles Playwright + Chromium)
Optional LLM API keys for LLM-based extraction

Implementation Details

curl -X POST https:///crawl \
  -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"], "crawler_config": {"type": "CrawlerRunConfig", "params": {"cache_mode": "bypass"}}}'

Interactive API docs live at https:///docs, and the built-in playground at /playground. The MCP endpoint for Claude Desktop / Cursor is available at /mcp/sse.

Why Deploy Crawl4AI on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Crawl4AI on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.