Scraper.is MCP
A Model Context Protocol (MCP) integration for Scraper.is - A powerful web scraping tool for AI assistants.
This package allows AI assistants to scrape web content through the MCP protocol, enabling them to access up-to-date information from the web.
Features
- 🌐 Web Scraping: Extract content from any website
- 📸 Screenshots: Capture visual representations of web pages
- 📄 Multiple Formats: Get content in markdown, HTML, or JSON
- 🔄 Progress Updates: Real-time progress reporting during scraping operations
- 🔌 MCP Integration: Seamless integration with MCP-compatible AI assistants
Installation
Installing via Smithery
To install scaperis-mcp for Claude Desktop automatically via Smithery:
npx -y @smithery/cli install @Ai-Quill/scaperis-mcp --client claude
Manual Installation
npm install -g scraperis-mcp
Or with yarn:
yarn global add scraperis-mcp
Prerequisites
You need a Scraper.is API key to use this package.
Getting Your API Key
- Sign up or log in at scraper.is
- Navigate to the API Keys section in your dashboard: https://www.scraper.is/dashboard/apikeys
- Create a new API key or copy your existing key
- Store this key securely as you'll need it to use this package
Usage
Environment Setup
Create a .env file with your Scraper.is API key:
SCRAPERIS_API_KEY=your_api_key_here
Claude Desktop Integration
To use this package with Claude Desktop:
-
Install the package globally:
npm install -g scraperis-mcp -
Add the following configuration to your
claude_desktop_config.jsonfile:{ "mcpServers": { "scraperis_scraper": { "command": "scraperis-mcp", "args": [], "env": { "SCRAPERIS_API_KEY": "your-api-key-here", "DEBUG": "*" } } } } -
Replace
your-api-key-herewith your actual Scraper.is API key. -
Restart Claude Desktop to apply the changes.
Running with MCP Inspector
For development and testing, you can use the MCP Inspector:
npx @modelcontextprotocol/inspector scraperis-mcp
Integration with AI Assistants
This package is designed to be used with AI assistants that support the Model Context Protocol (MCP). When properly configured, the AI assistant can use the following tools:
Scrape Tool
The scrape tool allows the AI to extract content from websites. It supports various formats:
markdown: Returns the content in markdown formathtml: Returns the content in HTML formatscreenshot: Returns a screenshot of the webpagejson: Returns structured data in JSON format
Example prompt for the AI:
Can you scrape the latest news from techcrunch.com and summarize it for me?
API Reference
Tools
scrape
Scrapes content from a webpage based on a prompt.
Parameters:
prompt(string): The prompt describing what to scrape, including the URLformat(string): The format to return the content in (markdown,html,screenshot,json,quick)
Example:
{
"prompt": "Get me the top 10 products from producthunt.com",
"format": "markdown"
}
Development
Setup
-
Clone the repository:
git clone https://github.com/Ai-Quill/scraperis-mcp.git cd scraperis-mcp -
Install dependencies:
npm install -
Build the project:
npm run build
Scripts
npm run build: Build the projectnpm run watch: Watch for changes and rebuildnpm run dev: Run with MCP Inspector for developmentnpm run test: Run testsnpm run lint: Run ESLint
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
License
This project is licensed under the MIT License - see the LICENSE file for details.