MCP Web URL Reader
An MCP server that exposes a single tool, read_web_url, which fetches absolute URLs with curl -sL after prepending a configurable CUSTOM_PREFIX on the server side. This server uses the MCP TypeScript/JavaScript SDK and serves a Streamable HTTP endpoint at /mcp for compatibility across many AI applications. Perfect for routing AI web requests through proxies, caching layers, or custom gateways.
Features
- 🌐 Simple web URL reading via curl
- 🔧 Configurable URL prefix for proxying/routing
- 🐳 Tiny Docker image based on Node.js
- 🚀 Easy deployment with Docker Compose
- 🛠️ Compatible with any MCP-enabled AI application
How It Works
The AI thinks it's reading a normal URL:
curl -sL https://www.example.com
But the server actually executes:
curl -sL CUSTOM_PREFIX/https://www.example.com
This allows you to:
- Route requests through a caching proxy
- Add authentication layers
- Use custom gateways
- Bypass rate limits with your own infrastructure
- Monitor/log AI web access
Quick Start
Using Docker Compose
See docker-compose.yml for a working example service mapping 8080:8080 and injecting both environment variables.
The MCP server will be available at http://localhost:8080
Environment Variables
| Variable | Description | Default |
|---|---|---|
CUSTOM_PREFIX | URL prefix to prepend to all requests | "" (empty) |
INTERNAL_PORT | Internal server port | 80800 |
Configuration Examples
Direct Mode (No Prefix)
environment:
- CUSTOM_PREFIX=
- PORT=3000
Proxy Mode
environment:
- CUSTOM_PREFIX=https://proxy.myserver.com
- INTERNAL_PORT=8080
Cache Gateway
environment:
- CUSTOM_PREFIX=http://cache-server:8000/fetch
- INTERNAL_PORT=8080
Using with AI Applications, MCP Client Integration
The server exposes tools over Streamable HTTP: initialize a session with a POST to /mcp, reuse Mcp-Session-Id from the response, then call tools/call with { name: "read_web_url", arguments: { url: "https://example.com" } }. The server will run curl -sL CUSTOM_PREFIX/https://example.com and return text content. Ensure the Mcp-Session-Id header is included in subsequent requests and allowed via CORS.
Claude Desktop (MCP)
Add to your claude_desktop_config.json:
{
"mcpServers": {
"web-url-reader": {
"url": "http://localhost:8080/mcp"
}
}
}
n8n
Use the MCP Server node and point it to http://localhost:8080/mcp
Custom Applications
Connect to the MCP endpoint at http://localhost:8080/mcp
Tool Schema
The server exposes one tool:
read_web_url
- Description: Read and return the contents of a web page
- Parameter:
url(string, required) - The complete URL to fetch - Returns: Text content of the web page
Example tool call:
{
"name": "read_web_url",
"arguments": {
"url": "https://www.example.com"
}
}
Use Cases
Caching Layer
Route all AI web requests through your caching server to reduce external calls:
CUSTOM_PREFIX=http://cache-server/proxy
Authentication Gateway
Add authentication to web requests:
CUSTOM_PREFIX=http://auth-gateway/fetch
Rate Limit Management
Route through your own infrastructure to manage rate limits:
CUSTOM_PREFIX=https://ratelimit-proxy.myserver.com
Privacy/Monitoring
Log and monitor all AI web access:
CUSTOM_PREFIX=http://monitoring-gateway/fetch
Run
- Open
http://localhost:8080/for a basic health probe and connect your MCP client tohttp://localhost:8080/mcp. - The MCP endpoint is on
http://localhost:8080/mcpwith Streamable HTTP semantics for POST/GET/DELETE and CORS exposure forMcp-Session-Id.
Troubleshooting
Server won't start
- Check that port 8080 is not in use
- Verify environment variables are set correctly
- Check Docker logs:
docker-compose logs -f
Requests failing
- Verify CUSTOM_PREFIX is accessible from container
- Test curl manually:
docker exec -it mcp-web-reader curl -sL YOUR_URL - Check timeout settings (default 30s)
GitHub Actions not building
- Ensure workflow permissions are enabled
- Check that GITHUB_TOKEN has package write access
- Verify Dockerfile syntax
License
MIT License - Feel free to use and modify as needed.
Contributing
Pull requests welcome! Please ensure:
- Docker image builds successfully
- README is updated for new features
- GitHub Actions workflow passes