---
title: "Deploy LightCrawl with Redis"
description: "Scalable LightCrawl with Redis backend for distributed queue-based crawling"
category: "AI/ML"
url: https://railway.com/deploy/lightcrawl-with-redis
---

# Deploy LightCrawl with Redis

Scalable LightCrawl with Redis backend for distributed queue-based crawling

**[Deploy LightCrawl with Redis on Railway](https://railway.com/template/lightcrawl-with-redis)**

- **Creator:** Yosuke Suzuki's Projects
- **Category:** AI/ML
- **Total deploys:** 1

## Template content

### Redis https://cdn.sanity.io/images/sy1jschh/production/0ce0bfdcfbdbf69662b1116671f97c2dd788b655-157x157.svg

- **Image:** redis:8.2.1
- **Start command:** `/bin/sh -c "rm -rf $RAILWAY_VOLUME_MOUNT_PATH/lost+found/ && exec docker-entrypoint.sh redis-server --requirepass $REDIS_PASSWORD --save 60 1 --dir $RAILWAY_VOLUME_MOUNT_PATH"`

### LightCrawl

- **Source:** yosuke1024/LightCrawl
- **Health check:** /health
- **Public domain:** Yes

## Documentation

# Deploy and Host LightCrawl with Redis on Railway

This template deploys **LightCrawl with Redis**, enabling queue-based distributed crawling and horizontal scaling. It acts as a highly scalable, self-hostable Web scraping API and Model Context Protocol (MCP) server that converts any web page into clean Markdown.

## About Hosting LightCrawl with Redis

Deploying this scalable setup on Railway takes less than a minute. Railway automatically provisions:
1. **LightCrawl Application**: A container built from the Dockerfile, containing Chromium for stealth crawling.
2. **Redis Database**: Used as a distributed queue and state backend for crawling tasks.

The template automatically links the Redis service to the LightCrawl container via the `REDIS_URL` environment variable. Upon startup, LightCrawl detects this variable and spins up background distributed workers. You can horizontally scale the application service to handle massive parallel scraping workloads.

You only need to configure basic variables: specifying `PORT`, setting `NODE_ENV` to production, and setting `API_KEY` (which Railway can generate automatically) to secure your endpoints.

## Common Use Cases

- **High-Volume Crawling**: Queue and process large websites concurrently using multiple worker instances coordinated via Redis.
- **LLM Context Injection**: Scrape target web pages and extract clean Markdown content to feed directly into LLM prompt contexts.
- **MCP Server for AI Agents**: Register LightCrawl as an MCP tool inside AI clients (Cursor, Claude Desktop) to let agents scrape the web in real-time.
- **Secure Sandboxed Scraping**: Run execution-heavy browser sessions inside an isolated Railway container to protect client environments.

## Dependencies for LightCrawl Hosting

- **Redis**: Coordinates distributed crawl queues and jobs across instances.
- **Playwright (Chromium)**: Headless browser engine used to render pages and bypass bot detection.
- **Mozilla Readability**: Content extraction engine to strip boilerplate code (menus, ads, footers) and return clean semantic content.

### Deployment Dependencies

- [GitHub Repository](https://github.com/yosuke1024/LightCrawl)
- [Model Context Protocol (MCP) Documentation](https://modelcontextprotocol.io/)

### Implementation Details

With Redis configured, crawl jobs are queued and processed asynchronously by worker instances:

```typescript
// If REDIS_URL is detected, LightCrawl activates the distributed queue system
if (process.env.REDIS_URL && process.env.ENABLE_DISTRIBUTED_WORKER !== 'false') {
  // Starts Redis distributed worker cooperatively
  await startRedisWorker();
}
```

## Why Deploy LightCrawl on Railway?
Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying LightCrawl on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.


## Similar templates

- [Chat Chat](https://railway.com/deploy/-WWW5r) — Chat Chat, your own unified chat and search to AI platform.
- [stella](https://railway.com/deploy/stella) — Self-host stella with web, API, Postgres, Redis, and object storage.
- [Hermes Agent | OpenClaw Alternative with Dashboard](https://railway.com/deploy/hermes-agent-or-openclaw-alternative-wit) — Self-Hosted Hermes AI Agent for Telegram, Discord & Slack

Open this page in a browser: https://railway.com/deploy/lightcrawl-with-redis