Deploy and Host LightCrawl with Redis on Railway

This template deploys LightCrawl with Redis, enabling queue-based distributed crawling and horizontal scaling. It acts as a highly scalable, self-hostable Web scraping API and Model Context Protocol (MCP) server that converts any web page into clean Markdown.

About Hosting LightCrawl with Redis

Deploying this scalable setup on Railway takes less than a minute. Railway automatically provisions:

LightCrawl Application: A container built from the Dockerfile, containing Chromium for stealth crawling.
Redis Database: Used as a distributed queue and state backend for crawling tasks.

The template automatically links the Redis service to the LightCrawl container via the REDIS_URL environment variable. Upon startup, LightCrawl detects this variable and spins up background distributed workers. You can horizontally scale the application service to handle massive parallel scraping workloads.

You only need to configure basic variables: specifying PORT, setting NODE_ENV to production, and setting API_KEY (which Railway can generate automatically) to secure your endpoints.

Common Use Cases

High-Volume Crawling: Queue and process large websites concurrently using multiple worker instances coordinated via Redis.
LLM Context Injection: Scrape target web pages and extract clean Markdown content to feed directly into LLM prompt contexts.
MCP Server for AI Agents: Register LightCrawl as an MCP tool inside AI clients (Cursor, Claude Desktop) to let agents scrape the web in real-time.
Secure Sandboxed Scraping: Run execution-heavy browser sessions inside an isolated Railway container to protect client environments.

Dependencies for LightCrawl Hosting

Redis: Coordinates distributed crawl queues and jobs across instances.
Playwright (Chromium): Headless browser engine used to render pages and bypass bot detection.
Mozilla Readability: Content extraction engine to strip boilerplate code (menus, ads, footers) and return clean semantic content.

Deployment Dependencies

Implementation Details

With Redis configured, crawl jobs are queued and processed asynchronously by worker instances:

// If REDIS_URL is detected, LightCrawl activates the distributed queue system
if (process.env.REDIS_URL && process.env.ENABLE_DISTRIBUTED_WORKER !== 'false') {
  // Starts Redis distributed worker cooperatively
  await startRedisWorker();
}

Why Deploy LightCrawl on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying LightCrawl on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.