mcp-fetch
A minimal MCP (Model Context Protocol) server that exposes a single tool: fetch_page.
It supports:
- Browser automation via Playwright (dynamic content, scrolling, waiting for selectors)
- Basic anti-bot shaping (User-Agent, mouse movement, randomized request delays, proxy rotation)
- Large responses via chunking with a
transfer_idcursor - On-disk chunk cache so agents can request more chunks later
Run (uvx)
From this directory:
uvx --no-cache --refresh --from . mcp-fetch
This is an MCP stdio server: it typically does not print anything until an MCP client connects and sends requests. If you run it directly in a terminal, it should stay running and wait for stdin from an MCP host.
For interactive/manual testing, it will auto-run an HTTP server when stdin is a TTY:
- Open:
http://127.0.0.1:8000/mcp - Override:
MCP_FETCH_TRANSPORT=stdio|http,MCP_FETCH_PORT=8000 - Or flags:
mcp-fetch --transport http --port 8000
Client config note
If you use uvx, you must include the provided command name at the end. This works:
cmd: uvx.exe
args: ["--from", "E:\\Private\\Mcp\\fetch", "mcp-fetch", "--transport", "stdio"]
Tool: fetch_page
Typical flow:
- Call with
url(gets chunk 0 andtransfer_id) - If you need more, call again with
transfer_idandoffset=next_offset
Notes:
- This server allows private/internal network access by default (your requested behavior).
- Only
http://andhttps://URLs are allowed. - Use
headersfor request headers. - Set
to_markdown=falseto return raw HTML instead of Markdown.
Tests
Run unit tests:
python -m unittest discover -s tests -v
There is a Playwright integration test (dynamic DOM). It is disabled by default:
- Run with:
MCP_FETCH_RUN_PLAYWRIGHT_TESTS=1 python -m unittest discover -s tests -v
Configuration (env)
MCP_FETCH_CACHE_DIR(default:./.mcp-fetch-cache)MCP_FETCH_TTL_SECONDS(default:1800)MCP_FETCH_MAX_CACHE_BYTES_TOTAL(default:536870912) (512 MiB)MCP_FETCH_MAX_SINGLE_TRANSFER_BYTES(default:209715200) (200 MiB)MCP_FETCH_WAIT_CHUNK_TIMEOUT_SECONDS(default:30)MCP_FETCH_EXPOSE_SHUTDOWN(default: unset) Set to1to expose theshutdowntool.