Spider MCP Server
An MCP server for crawling and spidering documentation websites, extracting clean text content, and using LLM-powered analysis to provide intelligent summaries and context through the Model Context Protocol.
Features
- Crawl entire documentation websites with configurable depth limits
- Extract clean text content from HTML pages using multiple extraction methods
- Intelligent content parsing that removes navigation, ads, and other noise
- LLM-powered content analysis using Anthropic Claude (Haiku/Sonnet)
- Automatic content summarization and key point extraction
- Intelligent code example extraction and categorization
- Code example detection across multiple programming languages
- Intelligent link discovery and relevance ranking
- Content type classification (tutorial, reference, API docs, etc.)
- Respect robots.txt and implement proper crawling etiquette
- File-based caching with TTL and compression
- Search through cached documentation content with LLM enhancement
- Concurrent crawling with rate limiting
- Support for various documentation layouts and formats
Installation
git clone <repository-url>
cd spider-mcp
bun install
cp .env.example .env
Usage
As MCP Server
Run the server to expose MCP tools:
bun run dev
MCP Client Integration
This server implements the Model Context Protocol (MCP) specification and communicates via stdio transport. MCP enables AI assistants to securely access external tools and data sources. To use this server, you need an MCP-compatible client like Claude Desktop, or you can integrate it programmatically using the MCP SDK.
Claude Desktop Integration
Add to your Claude Desktop MCP configuration:
{
"mcpServers": {
"spider-mcp": {
"command": "bun",
"args": ["run", "/path/to/spider-mcp/src/index.ts"],
"env": {
"ANTHROPIC_API_KEY": "your_api_key_here"
}
}
}
}
Programmatic Usage Examples
// Example using @modelcontextprotocol/sdk
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
const client = new Client({
name: "spider-client",
version: "1.0.0"
}, {
capabilities: {}
});
// Connect to the spider MCP server
const transport = new StdioClientTransport({
command: "bun",
args: ["run", "/path/to/spider-mcp/src/index.ts"]
});
await client.connect(transport);
// Extract code examples from documentation
const result = await client.request({
method: "tools/call",
params: {
name: "spider_docs",
arguments: {
url: "https://docs.anthropic.com/en/docs/quickstart",
enable_llm_analysis: true,
llm_analysis_type: "code_examples",
max_depth: 2
}
}
});
Available Tools
spider_docs
Crawl a documentation website and cache the content with optional LLM analysis.
Parameters:
url(required): Base URL of the documentation sitemax_depth(optional): Maximum crawl depth (default: 3)include_patterns(optional): URL patterns to includeexclude_patterns(optional): URL patterns to excludeenable_llm_analysis(optional): Enable LLM-powered content analysisllm_analysis_type(optional): Type of analysis:full- Complete analysis with summary, key points, links, and code examplessummary- Content summarization onlylinks- Link analysis and relevance rankingclassification- Content type classificationcode_examples- Extract and categorize code examples only
Example with code extraction:
{
"url": "https://docs.example.com/api",
"max_depth": 2,
"enable_llm_analysis": true,
"llm_analysis_type": "code_examples"
}
get_page
Retrieve a specific page from the cache.
Parameters:
url(required): URL of the page to retrieve
search_docs
Search through cached documentation content.
Parameters:
query(required): Search querylimit(optional): Maximum results to return (default: 10)
list_pages
List all cached pages with optional filtering and sorting.
Parameters:
filter(optional): Filter pages by URL patternsort(optional): Sort field (url, title, timestamp)order(optional): Sort order (asc, desc)
clear_cache
Clear cached pages matching a pattern.
Parameters:
url_pattern(optional): Pattern to match for clearing
analyze_content
Perform LLM-powered analysis on a specific cached page.
Parameters:
url(required): URL of the page to analyzeanalysis_type(optional): Same options asspider_docsabove
Example for code extraction:
{
"url": "https://docs.example.com/api/users",
"analysis_type": "code_examples"
}
get_summary
Get an intelligent summary of a cached page.
Parameters:
url(required): URL of the page to summarizesummary_length(optional): Length of summary (short, medium, long)focus_areas(optional): Specific areas to focus on in the summary
Configuration
Configuration can be provided through:
- Environment variables (see
.env.example) - JSON configuration files in
config/directory - Runtime parameters passed to tools
LLM Integration
To enable LLM-powered content analysis, set your Anthropic API key:
export ANTHROPIC_API_KEY=your_api_key_here
The server will automatically detect the API key and enable LLM features. Without an API key, the server operates in basic mode with programmatic content extraction only.
Example Configuration
{
"maxDepth": 3,
"maxPages": 100,
"concurrency": 5,
"userAgent": "SpiderMCP/1.0 Documentation Crawler",
"timeout": 10000,
"retryAttempts": 3,
"cacheTTL": 86400000,
"respectRobotsTxt": true,
"includePatterns": ["/docs/*", "/api/*"],
"excludePatterns": ["/blog/*", "*.pdf"]
}
Development
bun test
bun run typecheck
bun run build
Testing the Spider
Use the included test script to try out the functionality locally:
# Test with code example extraction on a documentation site
bun run test-spider.ts https://docs.anthropic.com/en/docs/quickstart
# Test with a specific site focusing on code examples
ANTHROPIC_API_KEY=your_key bun run test-spider.ts https://docs.example.com
# Test code-only extraction
bun run test-spider.ts https://docs.python.org/3/tutorial/
The test script demonstrates the spider functionality outside of MCP and will show you:
- Crawled pages with metadata
- Extracted code examples with languages and categories
- LLM analysis results including summaries and classifications
- Cache statistics and performance metrics
Note: The test script uses the spider functionality directly, not through MCP protocol.
Architecture
src/spider/- Core crawling and parsing logicsrc/mcp/- MCP server implementation and tool handlerssrc/extractors/- Content extraction strategiessrc/llm/- LLM integration and content analysissrc/utils/- Utility functions and configurationcache/- File system cache storageconfig/- Configuration files
Extraction Methods
The server supports multiple content extraction methods:
- Readability - Mozilla Readability algorithm for article extraction
- Cheerio - Custom CSS selector-based extraction
- Markdown - HTML to Markdown conversion with Turndown
- LLM Analysis - Anthropic Claude for intelligent content understanding and code extraction
Code Example Extraction
The LLM analysis can intelligently extract and categorize code examples from documentation:
- Language Detection: Automatically detects programming languages (JavaScript, Python, bash, JSON, etc.)
- Code Categories:
api_call- API requests, SDK calls, HTTP requestsconfiguration- Config files, settings, environment setupimplementation- Complete functions, classes, modulesusage_example- How to use a feature or librarysnippet- Small code fragments or utilitiescomplete_example- Full working examples or applications
- Format Preservation: Maintains exact formatting, indentation, and syntax
- Context Awareness: Provides meaningful descriptions for each code example
Example Output Format
When using code example extraction, the output includes structured code examples:
{
"codeExamples": [
{
"language": "javascript",
"code": "const response = await fetch('/api/users', {\n method: 'GET',\n headers: {\n 'Authorization': 'Bearer ' + apiKey\n }\n});",
"description": "Fetch users from API with authentication",
"category": "api_call"
},
{
"language": "python",
"code": "import requests\n\nresponse = requests.get('/api/users', headers={\n 'Authorization': f'Bearer {api_key}'\n})",
"description": "Python example for API authentication",
"category": "api_call"
},
{
"language": "json",
"code": "{\n \"database\": {\n \"host\": \"localhost\",\n \"port\": 5432\n }\n}",
"description": "Database configuration file",
"category": "configuration"
}
]
}
Caching
- Two-tier caching: in-memory + file system
- Configurable TTL and cache size limits
- Automatic cleanup of expired entries
- Domain-based cache organization
Robots.txt Compliance
- Fetches and parses robots.txt automatically
- Respects crawl-delay directives
- Honors disallow/allow rules
- Discovers sitemaps
Rate Limiting
- Configurable concurrent request limits
- Exponential backoff for failed requests
- Per-domain crawl delays
- Timeout handling
License
MIT