Deploy and Host WebSurfer with One Click on Railway

WebSurfer is an open-source, high-performance API designed to help Large Language Models (LLMs) surf the web. Built in Go, it fetches and processes web content with token-based content limiting, Redis caching, and API key authentication — making it easy to give your AI agents reliable, controlled access to live web data. It is a self-hosted alternative to proprietary web scraping APIs and LLM browsing tools.

With Railway, deploying WebSurfer becomes a true one-click experience. No server provisioning, no Redis setup, no reverse proxy configuration — just deploy and start fetching.

About Hosting WebSurfer on Railway (Self Hosting WebSurfer on Railway)

Self hosting WebSurfer traditionally means:

Provisioning a server with Go installed or Docker configured
Setting up and managing a Redis instance for caching
Configuring API keys, ports, and environment variables
Managing networking between WebSurfer and Redis
Handling service restarts, updates, and uptime monitoring

WebSurfer on Railway removes this burden completely. Railway automatically provisions both the WebSurfer container and a Redis instance, wires them together with internal networking, manages environment variables, and keeps both services running.

Why Deploy WebSurfer on Railway

Deploying WebSurfer on Railway is ideal for developers building AI agents, RAG pipelines, or LLM-powered tools that need live web access.

You get:

Automatic provisioning of WebSurfer + Redis
Internal networking between services
Secure environment variable management
One-click deployment with zero configuration
Built-in logs, metrics, and restart policies

Railway vs DigitalOcean

On DigitalOcean, you must create a droplet, install Docker, run a Redis container, configure networking between services, and manage firewall rules.

Railway vs Linode

Linode provides raw servers where you are responsible for OS updates, Docker installation, Redis management, and security patches. Railway abstracts these operational tasks and provides a ready-to-use WebSurfer instance with Redis instantly.

Railway vs Hetzner

Hetzner offers excellent performance and low costs but assumes strong sysadmin skills for Docker, Redis, and multi-service orchestration. Railway prioritizes ease, speed, and reliability, making WebSurfer ideal for rapid AI infrastructure deployment.

Common Use Cases

LLM Web Browsing: Give AI agents the ability to fetch and read live web pages with token-limited responses optimized for LLM context windows.
RAG Pipeline Data Ingestion: Fetch web content for Retrieval-Augmented Generation workflows, with caching to avoid redundant requests.
AI Research Assistants: Power research tools that pull real-time information from the web for summarization, fact-checking, or analysis.
Content Monitoring: Periodically fetch and process web pages to detect changes, track competitors, or monitor news sources.
API Gateway for Web Content: Provide a centralized, authenticated API endpoint that multiple AI services can use to access web content safely.

Dependencies for WebSurfer Hosted on Railway

WebSurfer has a minimal dependency footprint.

Core dependencies include:

Runtime: Go binary (compiled in Docker)
Cache: Redis for response caching and performance
Networking: HTTP server on configurable port
Configuration: YAML config file for site rules, rate limits, and cache TTLs

Deployment Dependencies

When deployed via Railway, WebSurfer automatically provisions:

WebSurfer container from the GitHub repo
Redis instance with internal networking
Environment variable binding for REDIS_URL
Port binding on 8080

Useful links:

WebSurfer on GitHub

Implementation Details

Key environment variables:

ADDR — Server address (default :8080)
REDIS_URL — Redis connection string (auto-configured by Railway)
CONFIG_FILE — Path to configuration YAML
LOG_LEVEL — Logging verbosity (debug, info, warn, error)

API endpoints:

POST /v1/fetch — Fetch and process a URL. Requires Bearer token or X-API-Key header. Accepts url and max_tokens parameters.
GET /health — Health check endpoint (no authentication required).

Example request:

curl -X POST https://your-websurfer.railway.app/v1/fetch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "max_tokens": 4000}'

Response includes: status code, page title, token estimate, content, cache state, and pagination metadata.

How Does WebSurfer Compare to Other Web Fetching Tools

WebSurfer vs Firecrawl

Firecrawl is a full-featured web scraping and crawling platform with JavaScript rendering, sitemap crawling, and structured extraction. WebSurfer is intentionally simpler — focused specifically on LLM-optimized content fetching with token limiting, making it lighter and faster for AI agent use cases.

WebSurfer vs Jina Reader API

Jina Reader is a hosted service that converts URLs to LLM-friendly text. WebSurfer gives you the same capability but fully self-hosted, with no API rate limits imposed by a third party and full control over caching and configuration.

WebSurfer vs Browserless

Browserless provides headless Chrome as a service for full browser rendering. WebSurfer is lighter weight — it fetches and processes content without a full browser, making it faster and cheaper to run for content that does not require JavaScript rendering.

How to Use WebSurfer

Deploy: Click the Railway deploy button to launch WebSurfer and Redis with a single click.
Set API Key: Configure your API key environment variable in the Railway dashboard to secure the fetch endpoint.
Get URL: Copy the Railway-generated public URL for your WebSurfer instance.
Fetch Content: Send POST requests to /v1/fetch with a URL and max_tokens parameter to retrieve LLM-optimized web content.

How to Self Host WebSurfer on Other VPS

To self host WebSurfer on a traditional VPS:

Provision a server with Docker installed
Start a Redis instance:

docker run -d --name redis -p 6379:6379 redis:7-alpine

Clone the repository:

git clone https://github.com/joeychilson/websurfer.git
cd websurfer

Build and run the container:

docker build -t websurfer .
docker run -d \
  --name websurfer \
  -p 8080:8080 \
  -e REDIS_URL=redis://redis:6379 \
  -e ADDR=:8080 \
  --link redis \
  websurfer

Configure firewall rules to allow port 8080
Set up SSL/TLS termination with a reverse proxy
Configure API key authentication

With Railway and the WebSurfer template, all of this is handled automatically.

Features of WebSurfer

High-performance URL fetching optimized for LLM consumption
Token-based content limiting to fit LLM context windows
Redis caching for fast repeated requests and reduced bandwidth
API key authentication via Bearer token or X-API-Key header
Rate limiting and retry mechanisms for reliable fetching
Site-specific configuration rules via YAML
Lightweight Go binary with minimal resource usage

Official Pricing of WebSurfer

WebSurfer is completely free and open source. There are no licensing fees, usage limits, or premium tiers. You only pay for the infrastructure running it.

System Requirements

CPU: 1 vCPU (minimum)
RAM: 256 MB–512 MB for WebSurfer, 256 MB+ for Redis
Storage: Minimal (Redis uses memory-based caching)
Port: 8080 (API server)

FAQs

What is WebSurfer? WebSurfer is an open-source API built in Go that helps LLMs fetch and process web content. It provides token-limited responses, Redis caching, and API key authentication for AI agent web access.

Is WebSurfer free and open source? Yes. WebSurfer is fully open source under a permissive license with no fees. You only pay for the server or platform where you host it.

How do I self host WebSurfer? You can deploy WebSurfer with one click on Railway using the template, or run it manually via Docker on any VPS alongside a Redis instance.

Is WebSurfer suitable for production use? Yes. WebSurfer includes API key authentication, rate limiting, retry mechanisms, and Redis caching — making it production-ready for AI workflows.

How much does it cost to host WebSurfer on Railway? Typically $5–$10 per month for the WebSurfer container and Redis instance. Railway bills based on actual resource consumption.

Does WebSurfer render JavaScript? No. WebSurfer fetches and processes HTML content without a headless browser. For JavaScript-heavy sites, consider pairing it with a rendering service.

Can I use WebSurfer with any LLM? Yes. WebSurfer is LLM-agnostic — it returns processed web content via a standard REST API that any LLM framework can consume.

What is the max_tokens parameter? It controls how much content WebSurfer returns from a fetched page, measured in estimated LLM tokens. This helps you stay within your model's context window limits.

Does WebSurfer support authentication for fetched sites? WebSurfer supports custom headers and site-specific configuration via config.yaml, which can include custom user agents and request parameters.

What is the best open-source alternative to Jina Reader? WebSurfer is a strong self-hosted alternative to Jina Reader, offering the same LLM-optimized web fetching with full control over caching, rate limits, and configuration.

Ready to Deploy WebSurfer?

Click below to Deploy WebSurfer on Railway and give your AI agents reliable web access in minutes.

Why Deploy WebSurfer on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.