Deploy Web Tools (Open Source Alternative To Firecrawl, Linkup, Tavily, Exa or Bright Data)

Deploy and Host Web Tools on Railway

Web Tools is an open-source web toolkit that gives AI agents fourteen tools to search, fetch, screenshot, crawl, and archive the web. Available as an MCP server, REST API, and CLI. It consumes zero LLM tokens for web access, so your models spend their budget on reasoning, not searching. The web has always been free for humans, so why should AI agents have to pay per query?

About Hosting Web Tools

This template deploys a complete self-hosted web toolkit as six services on Railway: Redis, SearXNG (privacy-respecting metasearch engine), Crawl4AI (headless browser for content extraction, screenshots, PDFs, and JS execution), Scrapling (stealth fetching: residential egress and JS-challenge solving), Camoufox (stealth Firefox on a geo-targeted residential exit, for sources that refuse anything else), and the Web Tools Server that ties them together. An API key is auto-generated at deploy time to secure your endpoint. Once deployed, any MCP-compatible client (Claude Code, Claude Desktop, Cursor, Windsurf, etc.) can connect over HTTP and use all fourteen tools. A REST API (POST /api/v0/{tool_name}) is also available for non-MCP integrations. No per-query fees, no third-party API keys, no usage limits. You own the infrastructure and the data never leaves your stack.

Common Use Cases

• Replace paid search APIs: Drop-in replacement for Firecrawl, Linkup, Tavily, Exa, or Bright Data. Get web search, page fetching, and content extraction without per-query costs
• Supercharge AI coding agents: Connect Claude Code or Cursor to self-hosted web search and page fetching. Replace their built-in WebSearch and WebFetch tools so every search is private and free
• Web research and monitoring: Search the web, fetch pages as clean markdown, take screenshots, generate PDFs, execute JavaScript on pages, and query the Wayback Machine for historical snapshots
• Build custom integrations: Use the REST API to integrate web tools into any application or workflow

Dependencies for Web Tools Hosting

• Redis (7-alpine): In-memory cache used by SearXNG for rate limiting and result caching
• SearXNG: Privacy-respecting metasearch engine that aggregates results from Google, Brave, DuckDuckGo, and more. Builds from services/searxng/Dockerfile with optional PROXY_URL support for outgoing requests
• Crawl4AI: Headless browser service for crawling, screenshots, PDFs and JavaScript execution, and for rendering HTML to markdown. Note Crawl4AI >= 0.9 refuses proxy_config from a request body, so it always egresses on its own IP. Pin the image rather than tracking :latest
• Scrapling: Stealth fetch sidecar that serves web_fetch and web_html. Owns the rotating residential egress (for IP-reputation walls such as LinkedIn) and the JS-challenge solving (for Cloudflare-style walls) that Crawl4AI structurally cannot do. Builds from services/scrapling/Dockerfile; set PROXY_URL on it to enable mode=stealth
• Camoufox: Stealth Firefox sidecar on a geo-targeted residential exit, with a fingerprint whose locale and timezone derive from the exit IP. Serves the sources the other two cannot reach at all: ones that bot-gate datacenter IPs outright, or score the exit country as part of an anti-bot sensor decision. Also owns the two capabilities nothing else here has: a binary/PDF fetch through that exit (web_bytes) and warmed anti-bot sensor sessions (web_spa_fetch). Builds from services/camoufox/Dockerfile; set PROXY_URL (geo-targeted) and keep WORKERS=1
• Web Tools Server (Node.js 22): The HTTP server exposing MCP and REST API endpoints. Builds from the repo-root Dockerfile. Do not delete it; it is this service's build

Deployment Dependencies

• Web Tools GitHub Repository
• SearXNG Documentation
• Crawl4AI Documentation
• Model Context Protocol Specification

Implementation Details

The Web Tools Server exposes two interfaces:

MCP: Streamable HTTP endpoint at /mcp for MCP clients:

{
  "mcpServers": {
    "web_tools": {
      "type": "http",
      "url": "https://your-server.up.railway.app/mcp",
      "headers": {
        "Authorization": "Bearer your-api-key"
      }
    }
  }
}

REST API: Standard HTTP endpoints at /api/v0/{tool_name}:

curl -X POST https://your-server.up.railway.app/api/v0/web_search \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"query": "railway deployment"}'

The fourteen tools available are: web_search, web_fetch, web_html, web_screenshot, web_pdf, web_execute_js, web_crawl, web_bytes, web_eval, web_spa_fetch, web_recycle, web_snapshots, web_archive, and web_usage_stats.

Callers never choose a fetch engine. Which of the three browsers serves a URL, and whether it egresses through a residential proxy, in which country, or solves a JS challenge, is decided from the host inside the server. Adding a knob for it would put the burden of knowing which engine can reach which site on every caller.

Railway Service Configuration

Set Root Directory before connecting a subfolder service to the repo. Railway resolves a service's build config by walking up from its Root Directory, so a subfolder service without one inherits the repo root's Dockerfile, which is the Node server. The build then goes green and the container crashes on ZodError: API_KEY Required, because it is running the wrong program; and it repeats on every push, so a service deployed correctly by hand will replace itself later. railway updoes not fix it (it uploads the right files while the stored config still points at /), and the CLI cannot set the field, so use the dashboard or the API mutation documented in the README.

Scale the browsers by replicas, not workers. Camoufox keeps WORKERS=1: a warmed anti-bot session cannot be shared across processes. Use railway service scale --service camoufox eu-west=2, and note scale ADDS to the existing regions, so pass us-east=0 to move rather than spread. Otherwise you get replicas on two continents and a transatlantic round trip per request.

Give only the Web Tools Server a public domain. It is the authenticated front door; the other five talk over Railway's private network and should have no domain at all. SearXNG in particular has no authentication of its own, so a public domain makes it an open search proxy whose outgoing requests spend your metered PROXY_URL bandwidth. SEARXNG_SECRET_KEY does not change that, because it is an internal signing secret rather than a credential. CRAWL4AI_API_TOKEN is a credential, and it is the only thing between an exposed Crawl4AI and free use of your browser fleet. Removing the domain is what makes a service private; removing its credentials just makes it broken or open.

Wire the services together with reference variables rather than hardcoded *.railway.internal hostnames, so the wiring survives a rename and each port is only right in one place:

CRAWL4AI_URL       = http://${{Crawl4AI.RAILWAY_PRIVATE_DOMAIN}}:11235
CRAWL4AI_API_TOKEN = ${{Crawl4AI.CRAWL4AI_API_TOKEN}}
SCRAPLING_URL      = http://${{Scrapling.RAILWAY_PRIVATE_DOMAIN}}:${{Scrapling.PORT}}
CAMOUFOX_URL       = http://${{Camoufox.RAILWAY_PRIVATE_DOMAIN}}:${{Camoufox.PORT}}
SEARXNG_URL        = http://${{SearXNG.RAILWAY_PRIVATE_DOMAIN}}:8080

Reference ${{Service.PORT}} only where the service actually binds it and has no default of its own to diverge from. That holds for the two images in this repo: their CMD is uvicorn --port ${PORT} and they deliberately ship no ENV PORT, so the Railway variable is the single source of truth for both the bind and the URL, and a missing one stops the container at boot rather than yielding http://host:.

It does not hold for the two third-party images, whose URLs keep literal ports: SearXNG hardcodes --port 8080 in its entrypoint and Crawl4AI reads port: 11235from its own config.yml, so PORT is decoration on both and a reference to it is a guess that fails open. Crawl4AI's read 8000 while the app listened on 11235, which pointed every fetch at a closed port.

Service names are case-sensitive: ${{camoufox.…}} against a service named Camoufox resolves to an empty string rather than erroring, giving http://:8000.

Why Deploy Web Tools on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Web Tools on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.