Web Search MCP Server
An MCP server that provides Google search capabilities and web content viewing with advanced bot detection avoidance.
Features
- Google Custom Search with advanced filtering
- Web content viewing with markdown conversion
- Rate limiting and caching
- Browser instance pooling
- Bot detection avoidance using rebrowser-puppeteer
Prerequisites
- Bun runtime v1.0 or higher
- Google API credentials (API key and Search Engine ID)
Installation
# Install dependencies
bun install
# Build the TypeScript files
bun run build
Configuration
Cookie Setup
For authenticated site access, you'll need to:
- Install the Get cookies.txt LOCALLY Chrome extension
- Visit the sites you want to authenticate with and log in
- Use the extension to export your cookies in JSON format
- Store the exported cookies file in a secure location
- Set the
BROWSER_COOKIES_PATHenvironment variable to the absolute path of your cookies file
MCP Server Configuration
Add the server configuration to your MCP settings file:
- For Cline:
%APPDATA%\Code\User\globalStorage\rooveterinaryinc.roo-cline\settings\cline_mcp_settings.json - For Claude Desktop:
- MacOS/Linux:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json
- MacOS/Linux:
{
"mcpServers": {
"web-search": {
"command": "bun",
"args": [
"run",
"/ABSOLUTE/PATH/TO/web_search_mcp/dist/index.js"
],
"env": {
"GOOGLE_API_KEY": "your_api_key",
"GOOGLE_SEARCH_ENGINE_ID": "your_search_engine_id",
"MAX_CONCURRENT_BROWSERS": "3",
"BROWSER_TIMEOUT": "30000",
"RATE_LIMIT_WINDOW": "60000",
"RATE_LIMIT_MAX_REQUESTS": "60",
"SEARCH_CACHE_TTL": "3600",
"VIEW_URL_CACHE_TTL": "7200",
"MAX_CACHE_ITEMS": "1000",
"BROWSER_POOL_MIN": "1",
"BROWSER_POOL_MAX": "5",
"BROWSER_POOL_IDLE_TIMEOUT": "30000",
"REBROWSER_PATCHES_RUNTIME_FIX_MODE": "addBinding",
"REBROWSER_PATCHES_SOURCE_URL": "jquery.min.js",
"REBROWSER_PATCHES_UTILITY_WORLD_NAME": "util",
"REBROWSER_PATCHES_DEBUG": "0",
"BROWSER_COOKIES_PATH": "C:\\path\\to\\cookies.json",
"LOG_LEVEL": "info",
"NO_COLOR": "0",
"BUN_FORCE_COLOR": "1",
"FORCE_COLOR": "1"
}
}
}
}
Replace /ABSOLUTE/PATH/TO/web_search_mcp with the absolute path to your server directory.
Logging Configuration
The following environment variables control logging behavior:
LOG_LEVEL: Sets the logging level (error, warn, info, debug). Default: infoNO_COLOR: Disables colored output when set to "1"BUN_FORCE_COLOR: Controls colored output in Bun runtime (set to "0" to disable)FORCE_COLOR: Controls colored output globally (set to "0" to disable)
Bot Detection Avoidance
This server uses rebrowser-puppeteer to avoid bot detection:
-
Runtime.Enable Leak Prevention:
- Uses the addBinding technique to avoid Runtime.Enable detection
- Works with web workers and iframes
- Maintains access to the main world context
-
Source URL Masking:
- Changes Puppeteer's sourceURL to look like a legitimate script
- Helps avoid detection of automation tools
-
Utility World Name:
- Uses a generic utility world name
- Prevents detection through world name patterns
-
Browser Launch Configuration:
- Disables automation flags
- Uses optimized Chrome arguments
- Configures viewport and window settings
Using with Claude Desktop
-
Make sure you have Claude Desktop installed and updated to the latest version
-
Open your Claude Desktop configuration file:
- MacOS/Linux:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json
- MacOS/Linux:
-
Add the server configuration as shown in the Configuration section above.
-
Restart Claude Desktop
-
Look for the hammer icon
to confirm the tools are available
Available Tools
1. Search Tool
{
name: "search",
params: {
query: string;
trustedDomains?: string[];
excludedDomains?: string[];
resultCount?: number;
safeSearch?: boolean;
dateRestrict?: string;
}
}
2. View URL Tool
{
name: "view_url",
params: {
url: string;
includeImages?: boolean;
includeVideos?: boolean;
preserveLinks?: boolean;
formatCode?: boolean;
}
}
Troubleshooting
Claude Desktop Integration Issues
-
Check the logs:
# MacOS/Linux tail -n 20 -f ~/Library/Logs/Claude/mcp*.log # Windows type %APPDATA%\Claude\Logs\mcp*.log -
Common issues:
- Server not showing up: Check configuration file syntax and paths
- Tool calls failing: Check server logs and restart Claude Desktop
- Path issues: Ensure you're using absolute paths
For more detailed troubleshooting, refer to the MCP debugging guide.
Development
# Run in development mode with watch
bun --watch run dev
# Run tests
bun run test
# Run linter
bun run lint
Important Notes
-
Bot Detection:
- The bot detection avoidance features help prevent most common detection methods
- However, additional measures like proper proxies and user agents may be needed
- Some websites may still detect automation through other means
-
Performance:
- Browser instances are pooled and reused
- Idle browsers are automatically cleaned up
- Resource limits prevent overloading
License
MIT