MCP Deep Web Research Server (v0.3.0)

A Model Context Protocol (MCP) server for advanced web research.

Latest Changes

Added visit_page tool for direct webpage content extraction
Optimized performance to work within MCP timeout limits
- Reduced default maxDepth and maxBranching parameters
- Improved page loading efficiency
- Added timeout checks throughout the process
- Enhanced error handling for timeouts

This project is a fork of mcp-webresearch by mzxrai, enhanced with additional features for deep web research capabilities. We're grateful to the original creators for their foundational work.

Bring real-time info into Claude with intelligent search queuing, enhanced content extraction, and deep research capabilities.

Features

Intelligent Search Queue System
- Batch search operations with rate limiting
- Queue management with progress tracking
- Error recovery and automatic retries
- Search result deduplication
Enhanced Content Extraction
- TF-IDF based relevance scoring
- Keyword proximity analysis
- Content section weighting
- Readability scoring
- Improved HTML structure parsing
- Structured data extraction
- Better content cleaning and formatting
Core Features
- Google search integration
- Webpage content extraction
- Research session tracking
- Markdown conversion with improved formatting

Prerequisites

Node.js >= 18 (includes npm and npx)
Claude Desktop app

Installation

Installing via Smithery

To install Deep Web Research Server for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @PedroDnT/mcp-deepwebresearch --client claude

Global Installation (Recommended)

# Install globally using npm
npm install -g mcp-deepwebresearch

# Or using yarn
yarn global add mcp-deepwebresearch

# Or using pnpm
pnpm add -g mcp-deepwebresearch

Local Project Installation

# Using npm
npm install mcp-deepwebresearch

# Using yarn
yarn add mcp-deepwebresearch

# Using pnpm
pnpm add mcp-deepwebresearch

Claude Desktop Integration

After installing the package, add this entry to your claude_desktop_config.json:

Windows

{
  "mcpServers": {
    "deepwebresearch": {
      "command": "mcp-deepwebresearch",
      "args": []
    }
  }
}

Location: %APPDATA%\Claude\claude_desktop_config.json

macOS

{
  "mcpServers": {
    "deepwebresearch": {
      "command": "mcp-deepwebresearch",
      "args": []
    }
  }
}

Location: ~/Library/Application Support/Claude/claude_desktop_config.json

This config allows Claude Desktop to automatically start the web research MCP server when needed.

First-time Setup

After installation, run this command to install required browser dependencies:

npx playwright install chromium

Usage

Simply start a chat with Claude and send a prompt that would benefit from web research. If you'd like a prebuilt prompt customized for deeper web research, you can use the agentic-research prompt that we provide through this package. Access that prompt in Claude Desktop by clicking the Paperclip icon in the chat input and then selecting Choose an integration → deepwebresearch → agentic-research.

Tools

deep_research

Performs comprehensive research with content analysis

Arguments:

{
  topic: string;
  maxDepth?: number;      // default: 2
  maxBranching?: number;  // default: 3
  timeout?: number;       // default: 55000 (55 seconds)
  minRelevanceScore?: number;  // default: 0.7
}

Returns:

{
  findings: {
    mainTopics: Array<{name: string, importance: number}>;
    keyInsights: Array<{text: string, confidence: number}>;
    sources: Array<{url: string, credibilityScore: number}>;
  };
  progress: {
    completedSteps: number;
    totalSteps: number;
    processedUrls: number;
  };
  timing: {
    started: string;
    completed?: string;
    duration?: number;
    operations?: {
      parallelSearch?: number;
      deduplication?: number;
      topResultsProcessing?: number;
      remainingResultsProcessing?: number;
      total?: number;
    };
  };
}

parallel_search
- Performs multiple Google searches in parallel with intelligent queuing
- Arguments: { queries: string[], maxParallel?: number }
- Note: maxParallel is limited to 5 to ensure reliable performance

visit_page

Visit a webpage and extract its content
Arguments: { url: string }

Returns:

{
  url: string;
  title: string;
  content: string;  // Markdown formatted content
}

Prompts

`agentic-research`

A guided research prompt that helps Claude conduct thorough web research. The prompt instructs Claude to:

Start with broad searches to understand the topic landscape
Prioritize high-quality, authoritative sources
Iteratively refine the research direction based on findings
Keep you informed and let you guide the research interactively
Always cite sources with URLs

Configuration Options

The server can be configured through environment variables:

MAX_PARALLEL_SEARCHES: Maximum number of concurrent searches (default: 5)
SEARCH_DELAY_MS: Delay between searches in milliseconds (default: 200)
MAX_RETRIES: Number of retry attempts for failed requests (default: 3)
TIMEOUT_MS: Request timeout in milliseconds (default: 55000)
LOG_LEVEL: Logging level (default: 'info')

Error Handling

Common Issues

Rate Limiting
- Symptom: "Too many requests" error
- Solution: Increase SEARCH_DELAY_MS or decrease MAX_PARALLEL_SEARCHES
Network Timeouts
- Symptom: "Request timed out" error
- Solution: Ensure requests complete within the 60-second MCP timeout
Browser Issues
- Symptom: "Browser failed to launch" error
- Solution: Ensure Playwright is properly installed (npx playwright install)

Debugging

This is beta software. If you run into issues:

Check Claude Desktop's MCP logs:

# On macOS
tail -n 20 -f ~/Library/Logs/Claude/mcp*.log

# On Windows
Get-Content -Path "$env:APPDATA\Claude\logs\mcp*.log" -Tail 20 -Wait

Enable debug logging:
```
export LOG_LEVEL=debug
```

Development

Setup

# Install dependencies
pnpm install

# Build the project
pnpm build

# Watch for changes
pnpm watch

# Run in development mode
pnpm dev

Testing

# Run all tests
pnpm test

# Run tests in watch mode
pnpm test:watch

# Run tests with coverage
pnpm test:coverage

Code Quality

# Run linter
pnpm lint

# Fix linting issues
pnpm lint:fix

# Type check
pnpm type-check

Contributing

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add some amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Coding Standards

Follow TypeScript best practices
Maintain test coverage above 80%
Document new features and APIs
Update CHANGELOG.md for significant changes
Follow semantic versioning

Performance Considerations

Use batch operations where possible
Implement proper error handling and retries
Consider memory usage with large datasets
Cache results when appropriate
Use streaming for large content

Requirements

Node.js >= 18
Playwright (automatically installed as a dependency)

Verified Platforms

macOS
Windows
Linux

License

MIT

Credits

This project builds upon the excellent work of mcp-webresearch by mzxrai. The original codebase provided the foundation for our enhanced features and capabilities.

Author

qpd-v

MCP Deep Web Research Server (v0.3.0)

A Model Context Protocol (MCP) server for advanced web research.

Latest Changes

Added visit_page tool for direct webpage content extraction
Optimized performance to work within MCP timeout limits
- Reduced default maxDepth and maxBranching parameters
- Improved page loading efficiency
- Added timeout checks throughout the process
- Enhanced error handling for timeouts

This project is a fork of mcp-webresearch by mzxrai, enhanced with additional features for deep web research capabilities. We're grateful to the original creators for their foundational work.

Bring real-time info into Claude with intelligent search queuing, enhanced content extraction, and deep research capabilities.

Features

Intelligent Search Queue System
- Batch search operations with rate limiting
- Queue management with progress tracking
- Error recovery and automatic retries
- Search result deduplication
Enhanced Content Extraction
- TF-IDF based relevance scoring
- Keyword proximity analysis
- Content section weighting
- Readability scoring
- Improved HTML structure parsing
- Structured data extraction
- Better content cleaning and formatting
Core Features
- Google search integration
- Webpage content extraction
- Research session tracking
- Markdown conversion with improved formatting

Prerequisites

Node.js >= 18 (includes npm and npx)
Claude Desktop app

Installation

Installing via Smithery

To install Deep Web Research Server for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @PedroDnT/mcp-deepwebresearch --client claude

Global Installation (Recommended)

# Install globally using npm
npm install -g mcp-deepwebresearch

# Or using yarn
yarn global add mcp-deepwebresearch

# Or using pnpm
pnpm add -g mcp-deepwebresearch

Local Project Installation

# Using npm
npm install mcp-deepwebresearch

# Using yarn
yarn add mcp-deepwebresearch

# Using pnpm
pnpm add mcp-deepwebresearch

Claude Desktop Integration

After installing the package, add this entry to your claude_desktop_config.json:

Windows

{
  "mcpServers": {
    "deepwebresearch": {
      "command": "mcp-deepwebresearch",
      "args": []
    }
  }
}

Location: %APPDATA%\Claude\claude_desktop_config.json

macOS

{
  "mcpServers": {
    "deepwebresearch": {
      "command": "mcp-deepwebresearch",
      "args": []
    }
  }
}

Location: ~/Library/Application Support/Claude/claude_desktop_config.json

This config allows Claude Desktop to automatically start the web research MCP server when needed.

First-time Setup

After installation, run this command to install required browser dependencies:

npx playwright install chromium

Usage

Tools

deep_research

Performs comprehensive research with content analysis

Arguments:

{
  topic: string;
  maxDepth?: number;      // default: 2
  maxBranching?: number;  // default: 3
  timeout?: number;       // default: 55000 (55 seconds)
  minRelevanceScore?: number;  // default: 0.7
}

Returns:

{
  findings: {
    mainTopics: Array<{name: string, importance: number}>;
    keyInsights: Array<{text: string, confidence: number}>;
    sources: Array<{url: string, credibilityScore: number}>;
  };
  progress: {
    completedSteps: number;
    totalSteps: number;
    processedUrls: number;
  };
  timing: {
    started: string;
    completed?: string;
    duration?: number;
    operations?: {
      parallelSearch?: number;
      deduplication?: number;
      topResultsProcessing?: number;
      remainingResultsProcessing?: number;
      total?: number;
    };
  };
}

parallel_search
- Performs multiple Google searches in parallel with intelligent queuing
- Arguments: { queries: string[], maxParallel?: number }
- Note: maxParallel is limited to 5 to ensure reliable performance

visit_page

Visit a webpage and extract its content
Arguments: { url: string }

Returns:

{
  url: string;
  title: string;
  content: string;  // Markdown formatted content
}

Prompts

`agentic-research`

A guided research prompt that helps Claude conduct thorough web research. The prompt instructs Claude to:

Start with broad searches to understand the topic landscape
Prioritize high-quality, authoritative sources
Iteratively refine the research direction based on findings
Keep you informed and let you guide the research interactively
Always cite sources with URLs

Configuration Options

The server can be configured through environment variables:

MAX_PARALLEL_SEARCHES: Maximum number of concurrent searches (default: 5)
SEARCH_DELAY_MS: Delay between searches in milliseconds (default: 200)
MAX_RETRIES: Number of retry attempts for failed requests (default: 3)
TIMEOUT_MS: Request timeout in milliseconds (default: 55000)
LOG_LEVEL: Logging level (default: 'info')

Error Handling

Common Issues

Rate Limiting
- Symptom: "Too many requests" error
- Solution: Increase SEARCH_DELAY_MS or decrease MAX_PARALLEL_SEARCHES
Network Timeouts
- Symptom: "Request timed out" error
- Solution: Ensure requests complete within the 60-second MCP timeout
Browser Issues
- Symptom: "Browser failed to launch" error
- Solution: Ensure Playwright is properly installed (npx playwright install)

Debugging

This is beta software. If you run into issues:

Check Claude Desktop's MCP logs:

# On macOS
tail -n 20 -f ~/Library/Logs/Claude/mcp*.log

# On Windows
Get-Content -Path "$env:APPDATA\Claude\logs\mcp*.log" -Tail 20 -Wait

Enable debug logging:
```
export LOG_LEVEL=debug
```

Development

Setup

# Install dependencies
pnpm install

# Build the project
pnpm build

# Watch for changes
pnpm watch

# Run in development mode
pnpm dev

Testing

# Run all tests
pnpm test

# Run tests in watch mode
pnpm test:watch

# Run tests with coverage
pnpm test:coverage

Code Quality

# Run linter
pnpm lint

# Fix linting issues
pnpm lint:fix

# Type check
pnpm type-check

Contributing

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add some amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Coding Standards

Follow TypeScript best practices
Maintain test coverage above 80%
Document new features and APIs
Update CHANGELOG.md for significant changes
Follow semantic versioning

Performance Considerations

Use batch operations where possible
Implement proper error handling and retries
Consider memory usage with large datasets
Cache results when appropriate
Use streaming for large content

Requirements

Node.js >= 18
Playwright (automatically installed as a dependency)

Verified Platforms

macOS
Windows
Linux

License

MIT

Credits

This project builds upon the excellent work of mcp-webresearch by mzxrai. The original codebase provided the foundation for our enhanced features and capabilities.

Author

qpd-v