LLM Researcher

A lightweight MCP (Model Context Protocol) server for LLM orchestration that provides efficient web content search and extraction capabilities. This CLI tool enables LLMs to search DuckDuckGo and extract clean, LLM-friendly content from web pages.

Built with TypeScript, tsup, and vitest for modern development experience.

Features

MCP Server Support: Provides Model Context Protocol server for LLM integration
Free Operation: Uses DuckDuckGo HTML endpoint (no API costs)
GitHub Code Search: Search GitHub repositories for code examples and implementation patterns
Smart Content Extraction: Playwright + @mozilla/readability for clean content
LLM-Optimized Output: Sanitized Markdown (h1-h3, bold, italic, links only)
Rate Limited: Respects DuckDuckGo with 1 req/sec limit
Cross-Platform: Works on macOS, Linux, and WSL
Multiple Modes: CLI, MCP server, search, direct URL, and interactive modes
Type Safe: Full TypeScript implementation with strict typing
Modern Tooling: Built with tsup bundler and vitest testing

Installation

Prerequisites

Node.js 20.0.0 or higher
No local Chrome installation required (uses Playwright's bundled Chromium)

Setup

# Clone or download the project
cd light-research-mcp

# Install dependencies (using pnpm)
pnpm install

# Build the project
pnpm build

# Install Playwright browsers
pnpm install-browsers

# Optional: Link globally for system-wide access
pnpm link --global

Usage

MCP Server Mode

Use as a Model Context Protocol server to provide search and content extraction tools to LLMs:

# Start MCP server (stdio transport)
llmresearcher --mcp

# The server provides these tools to MCP clients:
# - github_code_search: Search GitHub repositories for code
# - duckduckgo_web_search: Search the web with DuckDuckGo
# - extract_content: Extract detailed content from URLs

Setting up with Claude Code

# Add as an MCP server to Claude Code
claude mcp add light-research-mcp /path/to/light-research-mcp/dist/bin/llmresearcher.js --mcp

# Or with project scope for team sharing
claude mcp add light-research-mcp -s project /path/to/light-research-mcp/dist/bin/llmresearcher.js --mcp

# List configured servers
claude mcp list

# Check server status
claude mcp get light-research-mcp

MCP Tool Usage Examples

Once configured, you can use these tools in Claude:

> Search for React hooks examples on GitHub
Tool: github_code_search
Query: "useState useEffect hooks language:javascript"

> Search for TypeScript best practices
Tool: duckduckgo_web_search  
Query: "TypeScript best practices 2024"
Locale: us-en (or wt-wt for no region)

> Extract content from a search result
Tool: extract_content
URL: https://example.com/article-from-search-results

Command Line Interface

# Search mode - Search DuckDuckGo and interactively browse results
llmresearcher "machine learning transformers"

# GitHub Code Search mode - Search GitHub for code
llmresearcher -g "useState hooks language:typescript"

# Direct URL mode - Extract content from specific URL
llmresearcher -u https://example.com/article

# Interactive mode - Enter interactive search session
llmresearcher

# Verbose logging - See detailed operation logs
llmresearcher -v "search query"

# MCP Server mode - Start as Model Context Protocol server
llmresearcher --mcp

Development

Scripts

# Build the project
pnpm build

# Build in watch mode (for development)
pnpm dev

# Run tests
pnpm test

# Run tests in CI mode (single run)
pnpm test:run

# Type checking
pnpm type-check

# Clean build artifacts
pnpm clean

# Install Playwright browsers
pnpm install-browsers

Interactive Commands

When in search results view:

1-10: Select a result by number
b or back: Return to search results
open <n>: Open result #n in external browser
q or quit: Exit the program

When viewing content:

b or back: Return to search results
/<term>: Search for term within the extracted content
open: Open current page in external browser
q or quit: Exit the program

Configuration

Environment Variables

Create a .env file in the project root:

USER_AGENT=Mozilla/5.0 (compatible; LLMResearcher/1.0)
TIMEOUT=30000
MAX_RETRIES=3
RATE_LIMIT_DELAY=1000
CACHE_ENABLED=true
MAX_RESULTS=10

Configuration File

Create ~/.llmresearcherrc in your home directory:

{
  "userAgent": "Mozilla/5.0 (compatible; LLMResearcher/1.0)",
  "timeout": 30000,
  "maxRetries": 3,
  "rateLimitDelay": 1000,
  "cacheEnabled": true,
  "maxResults": 10
}

Configuration Options

Option	Default	Description
`userAgent`	`Mozilla/5.0 (compatible; LLMResearcher/1.0)`	User agent for HTTP requests
`timeout`	`30000`	Request timeout in milliseconds
`maxRetries`	`3`	Maximum retry attempts for failed requests
`rateLimitDelay`	`1000`	Delay between requests in milliseconds
`cacheEnabled`	`true`	Enable/disable local caching
`maxResults`	`10`	Maximum search results to display

Architecture

Core Components

MCPResearchServer (src/mcp-server.ts)
- Model Context Protocol server implementation
- Three main tools: github_code_search, duckduckgo_web_search, extract_content
- JSON-based responses for LLM consumption
DuckDuckGoSearcher (src/search.ts)
- HTML scraping of DuckDuckGo search results with locale support
- URL decoding for /l/?uddg= format links
- Rate limiting and retry logic
GitHubCodeSearcher (src/github-code-search.ts)
- GitHub Code Search API integration via gh CLI
- Advanced query support with language, repo, and file filters
- Authentication and rate limiting
ContentExtractor (src/extractor.ts)
- Playwright-based page rendering with resource blocking
- @mozilla/readability for main content extraction
- DOMPurify sanitization and Markdown conversion
CLIInterface (src/cli.ts)
- Interactive command-line interface
- Search result navigation
- Content viewing and text search
Configuration (src/config.ts)
- Environment and RC file configuration loading
- Verbose logging support

Content Processing Pipeline

MCP Server Mode

Search:
- DuckDuckGo: HTML endpoint → Parse results → JSON response with pagination
- GitHub: Code Search API → Format results → JSON response with code snippets
Extract: URL from search results → Playwright navigation → Content extraction
Process: @mozilla/readability → DOMPurify sanitization → Clean JSON output
Output: Structured JSON for LLM consumption

CLI Mode

Search: DuckDuckGo HTML endpoint → Parse results → Display numbered list
Extract: Playwright navigation → Resource blocking → JS rendering
Process: @mozilla/readability → DOMPurify sanitization → Turndown Markdown
Output: Clean Markdown with h1-h3, bold, italic, links only

Security Features

Resource Blocking: Prevents loading of images, CSS, fonts for speed and security
Content Sanitization: DOMPurify removes scripts, iframes, and dangerous elements
Limited Markdown: Only allows safe formatting elements (h1-h3, strong, em, a)
Rate Limiting: Respects DuckDuckGo's rate limits with exponential backoff

Examples

MCP Server Usage with Claude Code

1. GitHub Code Search

You: "Find React hook examples for state management"

Claude uses github_code_search tool:
{
  "query": "useState useReducer state management language:javascript",
  "results": [
    {
      "title": "facebook/react/packages/react/src/ReactHooks.js",
      "url": "https://raw.githubusercontent.com/facebook/react/main/packages/react/src/ReactHooks.js",
      "snippet": "function useState(initialState) {\n  return dispatcher.useState(initialState);\n}"
    }
  ],
  "pagination": {
    "currentPage": 1,
    "hasNextPage": true,
    "nextPageToken": "2"
  }
}

2. Web Search with Locale

You: "Search for Vue.js tutorials in Japanese"

Claude uses duckduckgo_web_search tool:
{
  "query": "Vue.js チュートリアル 入門",
  "locale": "jp-jp",
  "results": [
    {
      "title": "Vue.js入門ガイド",
      "url": "https://example.com/vue-tutorial",
      "snippet": "Vue.jsの基本的な使い方を学ぶチュートリアル..."
    }
  ]
}

3. Content Extraction

You: "Extract the full content from that Vue.js tutorial"

Claude uses extract_content tool:
{
  "url": "https://example.com/vue-tutorial",
  "title": "Vue.js入門ガイド",
  "extractedAt": "2024-01-15T10:30:00.000Z",
  "content": "# Vue.js入門ガイド\n\nVue.jsは...\n\n## インストール\n\n..."
}

CLI Examples

Basic Search

$ llmresearcher "python web scraping"

🔍 Search Results:
══════════════════════════════════════════════════

1. Python Web Scraping Tutorial
   URL: https://realpython.com/python-web-scraping-practical-introduction/
   Complete guide to web scraping with Python using requests and Beautiful Soup...

2. Web Scraping with Python - BeautifulSoup and requests
   URL: https://www.dataquest.io/blog/web-scraping-python-tutorial/
   Learn how to scrape websites with Python, Beautiful Soup, and requests...

══════════════════════════════════════════════════
Commands: [1-10] select result | b) back | q) quit | open <n>) open in browser

> 1

📥 Extracting content from: Python Web Scraping Tutorial

📄 Content:
══════════════════════════════════════════════════

**Python Web Scraping Tutorial**
Source: https://realpython.com/python-web-scraping-practical-introduction/
Extracted: 2024-01-15T10:30:00.000Z

──────────────────────────────────────────────────

# Python Web Scraping: A Practical Introduction

Web scraping is the process of collecting and parsing raw data from the web...

## What Is Web Scraping?

Web scraping is a technique to automatically access and extract large amounts...

══════════════════════════════════════════════════
Commands: b) back to results | /<term>) search in text | q) quit | open) open in browser

> /beautiful soup

🔍 Found 3 matches for "beautiful soup":
──────────────────────────────────────────────────
Line 15: Beautiful Soup is a Python library for parsing HTML and XML documents.
Line 42: from bs4 import BeautifulSoup
Line 67: soup = BeautifulSoup(html_content, 'html.parser')

Direct URL Mode

$ llmresearcher -u https://docs.python.org/3/tutorial/

📄 Content:
══════════════════════════════════════════════════

**The Python Tutorial**
Source: https://docs.python.org/3/tutorial/
Extracted: 2024-01-15T10:35:00.000Z

──────────────────────────────────────────────────

# The Python Tutorial

Python is an easy to learn, powerful programming language...

## An Informal Introduction to Python

In the following examples, input and output are distinguished...

Verbose Mode

$ llmresearcher -v "nodejs tutorial"

[VERBOSE] Searching: https://duckduckgo.com/html/?q=nodejs%20tutorial&kl=us-en
[VERBOSE] Response: 200 in 847ms
[VERBOSE] Parsed 10 results
[VERBOSE] Launching browser...
[VERBOSE] Blocking resource: https://example.com/style.css
[VERBOSE] Blocking resource: https://example.com/image.png
[VERBOSE] Navigating to page...
[VERBOSE] Page loaded in 1243ms
[VERBOSE] Processing content with Readability...
[VERBOSE] Readability extraction successful
[VERBOSE] Closing browser...

Testing

Running Tests

# Run tests in watch mode
pnpm test

# Run tests once (CI mode)
pnpm test:run

# Run tests with coverage
pnpm test -- --coverage

Test Coverage

The test suite includes:

Unit Tests: Individual component testing
- search.test.ts: DuckDuckGo search functionality, URL decoding, rate limiting
- extractor.test.ts: Content extraction, Markdown conversion, resource management
- config.test.ts: Configuration validation and environment handling
Integration Tests: End-to-end workflow testing
- integration.test.ts: Complete search-to-extraction workflows, error handling, cleanup

Test Features

Fast: Powered by vitest for quick feedback
Type-safe: Full TypeScript support in tests
Isolated: Each test cleans up its resources
Comprehensive: Covers search, extraction, configuration, and integration scenarios

Troubleshooting

Common Issues

"Browser not found" Error

pnpm install-browsers

Rate Limiting Issues

The tool automatically handles rate limiting with 1-second delays
If you encounter 429 errors, the tool will automatically retry with exponential backoff

Content Extraction Failures

Some sites may block automated access
The tool includes fallback extraction methods (main → body content)
Use verbose mode (-v) to see detailed error information

Permission Denied (Unix/Linux)

chmod +x bin/llmresearcher.js

Performance Optimization

The tool is optimized for speed:

Resource Blocking: Automatically blocks images, CSS, fonts
Network Idle: Waits for JavaScript to complete rendering
Content Caching: Supports local caching to avoid repeated requests
Minimal Dependencies: Uses lightweight, focused libraries

Development

Project Structure

light-research-mcp/
├── dist/                      # Built JavaScript files (generated)
│   ├── bin/
│   │   └── llmresearcher.js   # CLI entry point (executable)
│   └── *.js                   # Compiled TypeScript modules
├── src/                       # TypeScript source files
│   ├── bin.ts                 # CLI entry point
│   ├── index.ts               # Main LLMResearcher class
│   ├── mcp-server.ts          # MCP server implementation
│   ├── search.ts              # DuckDuckGo search implementation
│   ├── github-code-search.ts  # GitHub Code Search implementation
│   ├── extractor.ts           # Content extraction with Playwright
│   ├── cli.ts                 # Interactive CLI interface
│   ├── config.ts              # Configuration management
│   └── types.ts               # TypeScript type definitions
├── test/                      # Test files (vitest)
│   ├── search.test.ts         # Search functionality tests
│   ├── extractor.test.ts      # Content extraction tests
│   ├── config.test.ts         # Configuration tests
│   ├── mcp-locale.test.ts     # MCP locale functionality tests
│   ├── mcp-content-extractor.test.ts # MCP content extractor tests
│   └── integration.test.ts    # End-to-end integration tests
├── tsconfig.json              # TypeScript configuration
├── tsup.config.ts             # Build configuration
├── vitest.config.ts           # Test configuration
├── package.json
└── README.md

Dependencies

Runtime Dependencies

@modelcontextprotocol/sdk: Model Context Protocol server implementation
@mozilla/readability: Content extraction from HTML
cheerio: HTML parsing for search results
commander: CLI argument parsing
dompurify: HTML sanitization
dotenv: Environment variable loading
jsdom: DOM manipulation for server-side processing
playwright: Browser automation for JS rendering
turndown: HTML to Markdown conversion

Development Dependencies

typescript: TypeScript compiler
tsup: Fast TypeScript bundler
vitest: Fast unit test framework
@types/*: TypeScript type definitions

License

MIT License - see LICENSE file for details.

Contributing

Fork the repository
Create a feature branch
Make your changes
Add tests if applicable
Submit a pull request

Roadmap

Planned Features

Enhanced MCP Tools: Additional specialized search tools for documentation, APIs, etc.
Caching Layer: SQLite-based URL → Markdown caching with 24-hour TTL
Search Engine Abstraction: Support for Brave Search, Bing, and other engines
Content Summarization: Optional AI-powered content summarization
Export Formats: JSON, plain text, and other output formats
Batch Processing: Process multiple URLs from file input
SSE Transport: Support for Server-Sent Events MCP transport

Performance Improvements

Parallel Processing: Concurrent content extraction for multiple results
Smart Caching: Intelligent cache invalidation based on content freshness
Memory Optimization: Streaming content processing for large documents

LLM Researcher

Built with TypeScript, tsup, and vitest for modern development experience.

Features

MCP Server Support: Provides Model Context Protocol server for LLM integration
Free Operation: Uses DuckDuckGo HTML endpoint (no API costs)
GitHub Code Search: Search GitHub repositories for code examples and implementation patterns
Smart Content Extraction: Playwright + @mozilla/readability for clean content
LLM-Optimized Output: Sanitized Markdown (h1-h3, bold, italic, links only)
Rate Limited: Respects DuckDuckGo with 1 req/sec limit
Cross-Platform: Works on macOS, Linux, and WSL
Multiple Modes: CLI, MCP server, search, direct URL, and interactive modes
Type Safe: Full TypeScript implementation with strict typing
Modern Tooling: Built with tsup bundler and vitest testing

Installation

Prerequisites

Node.js 20.0.0 or higher
No local Chrome installation required (uses Playwright's bundled Chromium)

Setup

# Clone or download the project
cd light-research-mcp

# Install dependencies (using pnpm)
pnpm install

# Build the project
pnpm build

# Install Playwright browsers
pnpm install-browsers

# Optional: Link globally for system-wide access
pnpm link --global

Usage

MCP Server Mode

Use as a Model Context Protocol server to provide search and content extraction tools to LLMs:

# Start MCP server (stdio transport)
llmresearcher --mcp

# The server provides these tools to MCP clients:
# - github_code_search: Search GitHub repositories for code
# - duckduckgo_web_search: Search the web with DuckDuckGo
# - extract_content: Extract detailed content from URLs

Setting up with Claude Code

# Add as an MCP server to Claude Code
claude mcp add light-research-mcp /path/to/light-research-mcp/dist/bin/llmresearcher.js --mcp

# Or with project scope for team sharing
claude mcp add light-research-mcp -s project /path/to/light-research-mcp/dist/bin/llmresearcher.js --mcp

# List configured servers
claude mcp list

# Check server status
claude mcp get light-research-mcp

MCP Tool Usage Examples

Once configured, you can use these tools in Claude:

> Search for React hooks examples on GitHub
Tool: github_code_search
Query: "useState useEffect hooks language:javascript"

> Search for TypeScript best practices
Tool: duckduckgo_web_search  
Query: "TypeScript best practices 2024"
Locale: us-en (or wt-wt for no region)

> Extract content from a search result
Tool: extract_content
URL: https://example.com/article-from-search-results

Command Line Interface

# Search mode - Search DuckDuckGo and interactively browse results
llmresearcher "machine learning transformers"

# GitHub Code Search mode - Search GitHub for code
llmresearcher -g "useState hooks language:typescript"

# Direct URL mode - Extract content from specific URL
llmresearcher -u https://example.com/article

# Interactive mode - Enter interactive search session
llmresearcher

# Verbose logging - See detailed operation logs
llmresearcher -v "search query"

# MCP Server mode - Start as Model Context Protocol server
llmresearcher --mcp

Development

Scripts

# Build the project
pnpm build

# Build in watch mode (for development)
pnpm dev

# Run tests
pnpm test

# Run tests in CI mode (single run)
pnpm test:run

# Type checking
pnpm type-check

# Clean build artifacts
pnpm clean

# Install Playwright browsers
pnpm install-browsers

Interactive Commands

When in search results view:

1-10: Select a result by number
b or back: Return to search results
open <n>: Open result #n in external browser
q or quit: Exit the program

When viewing content:

b or back: Return to search results
/<term>: Search for term within the extracted content
open: Open current page in external browser
q or quit: Exit the program

Configuration

Environment Variables

Create a .env file in the project root:

USER_AGENT=Mozilla/5.0 (compatible; LLMResearcher/1.0)
TIMEOUT=30000
MAX_RETRIES=3
RATE_LIMIT_DELAY=1000
CACHE_ENABLED=true
MAX_RESULTS=10

Configuration File

Create ~/.llmresearcherrc in your home directory:

{
  "userAgent": "Mozilla/5.0 (compatible; LLMResearcher/1.0)",
  "timeout": 30000,
  "maxRetries": 3,
  "rateLimitDelay": 1000,
  "cacheEnabled": true,
  "maxResults": 10
}

Configuration Options

Option	Default	Description
`userAgent`	`Mozilla/5.0 (compatible; LLMResearcher/1.0)`	User agent for HTTP requests
`timeout`	`30000`	Request timeout in milliseconds
`maxRetries`	`3`	Maximum retry attempts for failed requests
`rateLimitDelay`	`1000`	Delay between requests in milliseconds
`cacheEnabled`	`true`	Enable/disable local caching
`maxResults`	`10`	Maximum search results to display

Architecture

Core Components

MCPResearchServer (src/mcp-server.ts)
- Model Context Protocol server implementation
- Three main tools: github_code_search, duckduckgo_web_search, extract_content
- JSON-based responses for LLM consumption
DuckDuckGoSearcher (src/search.ts)
- HTML scraping of DuckDuckGo search results with locale support
- URL decoding for /l/?uddg= format links
- Rate limiting and retry logic
GitHubCodeSearcher (src/github-code-search.ts)
- GitHub Code Search API integration via gh CLI
- Advanced query support with language, repo, and file filters
- Authentication and rate limiting
ContentExtractor (src/extractor.ts)
- Playwright-based page rendering with resource blocking
- @mozilla/readability for main content extraction
- DOMPurify sanitization and Markdown conversion
CLIInterface (src/cli.ts)
- Interactive command-line interface
- Search result navigation
- Content viewing and text search
Configuration (src/config.ts)
- Environment and RC file configuration loading
- Verbose logging support

Content Processing Pipeline

MCP Server Mode

Search:
- DuckDuckGo: HTML endpoint → Parse results → JSON response with pagination
- GitHub: Code Search API → Format results → JSON response with code snippets
Extract: URL from search results → Playwright navigation → Content extraction
Process: @mozilla/readability → DOMPurify sanitization → Clean JSON output
Output: Structured JSON for LLM consumption

CLI Mode

Search: DuckDuckGo HTML endpoint → Parse results → Display numbered list
Extract: Playwright navigation → Resource blocking → JS rendering
Process: @mozilla/readability → DOMPurify sanitization → Turndown Markdown
Output: Clean Markdown with h1-h3, bold, italic, links only

Security Features

Resource Blocking: Prevents loading of images, CSS, fonts for speed and security
Content Sanitization: DOMPurify removes scripts, iframes, and dangerous elements
Limited Markdown: Only allows safe formatting elements (h1-h3, strong, em, a)
Rate Limiting: Respects DuckDuckGo's rate limits with exponential backoff

Examples

MCP Server Usage with Claude Code

1. GitHub Code Search

You: "Find React hook examples for state management"

Claude uses github_code_search tool:
{
  "query": "useState useReducer state management language:javascript",
  "results": [
    {
      "title": "facebook/react/packages/react/src/ReactHooks.js",
      "url": "https://raw.githubusercontent.com/facebook/react/main/packages/react/src/ReactHooks.js",
      "snippet": "function useState(initialState) {\n  return dispatcher.useState(initialState);\n}"
    }
  ],
  "pagination": {
    "currentPage": 1,
    "hasNextPage": true,
    "nextPageToken": "2"
  }
}

2. Web Search with Locale

You: "Search for Vue.js tutorials in Japanese"

Claude uses duckduckgo_web_search tool:
{
  "query": "Vue.js チュートリアル 入門",
  "locale": "jp-jp",
  "results": [
    {
      "title": "Vue.js入門ガイド",
      "url": "https://example.com/vue-tutorial",
      "snippet": "Vue.jsの基本的な使い方を学ぶチュートリアル..."
    }
  ]
}

3. Content Extraction

You: "Extract the full content from that Vue.js tutorial"

Claude uses extract_content tool:
{
  "url": "https://example.com/vue-tutorial",
  "title": "Vue.js入門ガイド",
  "extractedAt": "2024-01-15T10:30:00.000Z",
  "content": "# Vue.js入門ガイド\n\nVue.jsは...\n\n## インストール\n\n..."
}

CLI Examples

Basic Search

$ llmresearcher "python web scraping"

🔍 Search Results:
══════════════════════════════════════════════════

1. Python Web Scraping Tutorial
   URL: https://realpython.com/python-web-scraping-practical-introduction/
   Complete guide to web scraping with Python using requests and Beautiful Soup...

2. Web Scraping with Python - BeautifulSoup and requests
   URL: https://www.dataquest.io/blog/web-scraping-python-tutorial/
   Learn how to scrape websites with Python, Beautiful Soup, and requests...

══════════════════════════════════════════════════
Commands: [1-10] select result | b) back | q) quit | open <n>) open in browser

> 1

📥 Extracting content from: Python Web Scraping Tutorial

📄 Content:
══════════════════════════════════════════════════

**Python Web Scraping Tutorial**
Source: https://realpython.com/python-web-scraping-practical-introduction/
Extracted: 2024-01-15T10:30:00.000Z

──────────────────────────────────────────────────

# Python Web Scraping: A Practical Introduction

Web scraping is the process of collecting and parsing raw data from the web...

## What Is Web Scraping?

Web scraping is a technique to automatically access and extract large amounts...

══════════════════════════════════════════════════
Commands: b) back to results | /<term>) search in text | q) quit | open) open in browser

> /beautiful soup

🔍 Found 3 matches for "beautiful soup":
──────────────────────────────────────────────────
Line 15: Beautiful Soup is a Python library for parsing HTML and XML documents.
Line 42: from bs4 import BeautifulSoup
Line 67: soup = BeautifulSoup(html_content, 'html.parser')

Direct URL Mode

$ llmresearcher -u https://docs.python.org/3/tutorial/

📄 Content:
══════════════════════════════════════════════════

**The Python Tutorial**
Source: https://docs.python.org/3/tutorial/
Extracted: 2024-01-15T10:35:00.000Z

──────────────────────────────────────────────────

# The Python Tutorial

Python is an easy to learn, powerful programming language...

## An Informal Introduction to Python

In the following examples, input and output are distinguished...

Verbose Mode

$ llmresearcher -v "nodejs tutorial"

[VERBOSE] Searching: https://duckduckgo.com/html/?q=nodejs%20tutorial&kl=us-en
[VERBOSE] Response: 200 in 847ms
[VERBOSE] Parsed 10 results
[VERBOSE] Launching browser...
[VERBOSE] Blocking resource: https://example.com/style.css
[VERBOSE] Blocking resource: https://example.com/image.png
[VERBOSE] Navigating to page...
[VERBOSE] Page loaded in 1243ms
[VERBOSE] Processing content with Readability...
[VERBOSE] Readability extraction successful
[VERBOSE] Closing browser...

Testing

Running Tests

# Run tests in watch mode
pnpm test

# Run tests once (CI mode)
pnpm test:run

# Run tests with coverage
pnpm test -- --coverage

Test Coverage

The test suite includes:

Unit Tests: Individual component testing
- search.test.ts: DuckDuckGo search functionality, URL decoding, rate limiting
- extractor.test.ts: Content extraction, Markdown conversion, resource management
- config.test.ts: Configuration validation and environment handling
Integration Tests: End-to-end workflow testing
- integration.test.ts: Complete search-to-extraction workflows, error handling, cleanup

Test Features

Fast: Powered by vitest for quick feedback
Type-safe: Full TypeScript support in tests
Isolated: Each test cleans up its resources
Comprehensive: Covers search, extraction, configuration, and integration scenarios

Troubleshooting

Common Issues

"Browser not found" Error

pnpm install-browsers

Rate Limiting Issues

The tool automatically handles rate limiting with 1-second delays
If you encounter 429 errors, the tool will automatically retry with exponential backoff

Content Extraction Failures

Some sites may block automated access
The tool includes fallback extraction methods (main → body content)
Use verbose mode (-v) to see detailed error information

Permission Denied (Unix/Linux)

chmod +x bin/llmresearcher.js

Performance Optimization

The tool is optimized for speed:

Resource Blocking: Automatically blocks images, CSS, fonts
Network Idle: Waits for JavaScript to complete rendering
Content Caching: Supports local caching to avoid repeated requests
Minimal Dependencies: Uses lightweight, focused libraries

Development

Project Structure

light-research-mcp/
├── dist/                      # Built JavaScript files (generated)
│   ├── bin/
│   │   └── llmresearcher.js   # CLI entry point (executable)
│   └── *.js                   # Compiled TypeScript modules
├── src/                       # TypeScript source files
│   ├── bin.ts                 # CLI entry point
│   ├── index.ts               # Main LLMResearcher class
│   ├── mcp-server.ts          # MCP server implementation
│   ├── search.ts              # DuckDuckGo search implementation
│   ├── github-code-search.ts  # GitHub Code Search implementation
│   ├── extractor.ts           # Content extraction with Playwright
│   ├── cli.ts                 # Interactive CLI interface
│   ├── config.ts              # Configuration management
│   └── types.ts               # TypeScript type definitions
├── test/                      # Test files (vitest)
│   ├── search.test.ts         # Search functionality tests
│   ├── extractor.test.ts      # Content extraction tests
│   ├── config.test.ts         # Configuration tests
│   ├── mcp-locale.test.ts     # MCP locale functionality tests
│   ├── mcp-content-extractor.test.ts # MCP content extractor tests
│   └── integration.test.ts    # End-to-end integration tests
├── tsconfig.json              # TypeScript configuration
├── tsup.config.ts             # Build configuration
├── vitest.config.ts           # Test configuration
├── package.json
└── README.md

Dependencies

Runtime Dependencies

@modelcontextprotocol/sdk: Model Context Protocol server implementation
@mozilla/readability: Content extraction from HTML
cheerio: HTML parsing for search results
commander: CLI argument parsing
dompurify: HTML sanitization
dotenv: Environment variable loading
jsdom: DOM manipulation for server-side processing
playwright: Browser automation for JS rendering
turndown: HTML to Markdown conversion

Development Dependencies

typescript: TypeScript compiler
tsup: Fast TypeScript bundler
vitest: Fast unit test framework
@types/*: TypeScript type definitions

License

MIT License - see LICENSE file for details.

Contributing

Fork the repository
Create a feature branch
Make your changes
Add tests if applicable
Submit a pull request

Roadmap

Planned Features

Enhanced MCP Tools: Additional specialized search tools for documentation, APIs, etc.
Caching Layer: SQLite-based URL → Markdown caching with 24-hour TTL
Search Engine Abstraction: Support for Brave Search, Bing, and other engines
Content Summarization: Optional AI-powered content summarization
Export Formats: JSON, plain text, and other output formats
Batch Processing: Process multiple URLs from file input
SSE Transport: Support for Server-Sent Events MCP transport

Performance Improvements

Parallel Processing: Concurrent content extraction for multiple results
Smart Caching: Intelligent cache invalidation based on content freshness
Memory Optimization: Streaming content processing for large documents