lilFetch

An MCP (Model Context Protocol) server that scrapes webpages using crawl4ai and Playwright for more robust scraping. Tested on CoPilot in VSCode but may work for others.

Features

Enables html and/or text scraping of one or more urls directly in your chat prompt. Use the scraped response in followup queries for market research summarization, context for new file creation, etc.
Leverages Playwright and a headless instance of Chromium to load JS heavy sites and web apps where basic #fetch and curl commands fall short.
Strong focus on minimal commands and configuration to install and get scraping.

Prerequisites

Before installing, ensure:

Node.js 14+: Download from nodejs.org or via Homebrew (brew install node on macOS).
Python 3.10+: Auto-detected during setup. Install from python.org or Homebrew (brew install python@3.12 on macOS). If using pyenv, set a 3.10+ version active (pyenv global 3.12.0).
First Run Time: Setup downloads ~200MB (Playwright browsers) and takes 1-2 minutes.

Installation

Install globally for use across workspaces, or restrict to local installation if you just want to test in this repo or enhance it further.

Option 1: Global Install

For running npx lilfetch from any directory (portable CLI).

Clone the Repo

git clone https://github.com/jphdevsf/lilfetch-mcp.git lilfetch-mcp
cd lilfetch-mcp

Install Globally
```
npm run global-install
```
- Sets up Python venv in ~/.lilfetch-venv (user-wide).

Configure in Any VS Code Workspace (add to .vscode/mcp.json or global MCP settings):

{
  "servers": {
    "lilFetch": {
      "type": "stdio",
      "command": "npx",
      "args": ["lilfetch"]
    }
  }
}

Test It

In new terminal window, run npx lilfetch to start MCP server.
In VS Code, prompt with something like...

Use lilFetch to scrape top news headlines from www.cnn.com and write to a markdown file in root of my repo.

Option 2: Local Install

For testing/extending in the repo.

Clone the Repo

git clone https://github.com/jphdevsf/lilfetch-mcp.git lilfetch-mcp
cd lilfetch-mcp

Install Locally
```
npm install
```
- Sets up ./node_modules/lilfetch/ and .bin/lilfetch.
- Python venv in repo .venv (local to this project).

MCP.json Workspace Configuration Navigate to .vscode/mcp.json (create if missing) and add:

{
  "servers": {
    "lilFetch": {
      "type": "stdio",
      "command": "node",
      "args": ["bin/lilfetch.js"]
    }
  }
}

Note: Ensure bin/lilfetch.js is executable: Run chmod +x bin/lilfetch.js in the terminal.

Test It
- In new terminal window, navigate to this repo and run npm run dev or ./node_modules/.bin/lilfetch.
- In VS Code, prompt with something like...
```
Use lilFetch to scrape top news headlines from www.cnn.com and write to a markdown file in root of my repo.
```

Uninstallation

To fully remove a global installation (including the npm package and Python virtual environment):

Global Uninstall

From the repo directory (or anywhere):

npm run global-uninstall

This runs npm uninstall -g lilfetch to remove the global npm package and binary.
Followed by rm -rf ~/.lilfetch-venv to delete the user-wide Python venv (including installed deps and Playwright browsers).
Warning: The rm -rf command is irreversible. It only affects the .lilfetch-venv directory in your home folder. Back up if needed (unlikely).

Manual Uninstall (Alternative)

Remove npm package:
```
npm uninstall -g lilfetch
```
Remove Python venv:
```
rm -rf ~/.lilfetch-venv
```
- On Windows: rmdir /s /q %USERPROFILE%\.lilfetch-venv

Local Uninstall

For local installs (e.g., after npm install):

npm uninstall
rm -rf .venv

This removes the local Node modules and repo-specific Python venv.

Verification

npm list -g --depth=0 (no lilfetch).
which lilfetch (empty).
ls ~/.lilfetch-venv (no such file).

For local: rm -rf node_modules .venv and verify no ./.bin/lilfetch.

Development

Edit mcp_server.py for Python logic.
Update bin/lilfetch.js for wrapper changes.
Bump version in package.json, then npm run pack.
For global testing: npm install -g . then npx lilfetch.

Troubleshooting

Permission Errors (Global Install): See Prerequisites for user-owned NPM setup. Avoid sudo—use the config steps.
Python Not Found/Version Error: Ensure Python 3.8+ is in PATH. For pyenv: pyenv install 3.12.0 && pyenv global 3.12.0, then re-run install. Check: python3 --version.
Venv/Deps Fail: For local: Delete .venv and re-run npm install. For global: Delete ~/.lilfetch-venv and re-run npm install -g .. Manual fix (local): python3 -m venv .venv && .venv/bin/pip install -r requirements.txt && .venv/bin/python -m playwright install. Manual fix (global): python3 -m venv ~/.lilfetch-venv && ~/.lilfetch-venv/bin/pip install -r requirements.txt && ~/.lilfetch-venv/bin/python -m playwright install.
Playwright Browsers Missing: Run python -m playwright install in the venv (or manually as logged).
MCP Not Detected in VS Code: Restart VS Code after config; ensure workspace is open correctly.
Uninstall:
- Global: npm uninstall -g lilfetch + rm -rf ~/.lilfetch-venv.
- Local: rm -rf node_modules package-lock.json .venv.

૮₍ • ᴥ • ₎ა

License: MIT

lilFetch

An MCP (Model Context Protocol) server that scrapes webpages using crawl4ai and Playwright for more robust scraping. Tested on CoPilot in VSCode but may work for others.

Features

Enables html and/or text scraping of one or more urls directly in your chat prompt. Use the scraped response in followup queries for market research summarization, context for new file creation, etc.
Leverages Playwright and a headless instance of Chromium to load JS heavy sites and web apps where basic #fetch and curl commands fall short.
Strong focus on minimal commands and configuration to install and get scraping.

Prerequisites

Before installing, ensure:

Node.js 14+: Download from nodejs.org or via Homebrew (brew install node on macOS).
Python 3.10+: Auto-detected during setup. Install from python.org or Homebrew (brew install python@3.12 on macOS). If using pyenv, set a 3.10+ version active (pyenv global 3.12.0).
First Run Time: Setup downloads ~200MB (Playwright browsers) and takes 1-2 minutes.

Installation

Install globally for use across workspaces, or restrict to local installation if you just want to test in this repo or enhance it further.

Option 1: Global Install

For running npx lilfetch from any directory (portable CLI).

Clone the Repo

git clone https://github.com/jphdevsf/lilfetch-mcp.git lilfetch-mcp
cd lilfetch-mcp

Install Globally
```
npm run global-install
```
- Sets up Python venv in ~/.lilfetch-venv (user-wide).

Configure in Any VS Code Workspace (add to .vscode/mcp.json or global MCP settings):

{
  "servers": {
    "lilFetch": {
      "type": "stdio",
      "command": "npx",
      "args": ["lilfetch"]
    }
  }
}

Test It

In new terminal window, run npx lilfetch to start MCP server.
In VS Code, prompt with something like...

Use lilFetch to scrape top news headlines from www.cnn.com and write to a markdown file in root of my repo.

Option 2: Local Install

For testing/extending in the repo.

Clone the Repo

git clone https://github.com/jphdevsf/lilfetch-mcp.git lilfetch-mcp
cd lilfetch-mcp

Install Locally
```
npm install
```
- Sets up ./node_modules/lilfetch/ and .bin/lilfetch.
- Python venv in repo .venv (local to this project).

MCP.json Workspace Configuration Navigate to .vscode/mcp.json (create if missing) and add:

{
  "servers": {
    "lilFetch": {
      "type": "stdio",
      "command": "node",
      "args": ["bin/lilfetch.js"]
    }
  }
}

Note: Ensure bin/lilfetch.js is executable: Run chmod +x bin/lilfetch.js in the terminal.

Test It
- In new terminal window, navigate to this repo and run npm run dev or ./node_modules/.bin/lilfetch.
- In VS Code, prompt with something like...
```
Use lilFetch to scrape top news headlines from www.cnn.com and write to a markdown file in root of my repo.
```

Uninstallation

To fully remove a global installation (including the npm package and Python virtual environment):

Global Uninstall

From the repo directory (or anywhere):

npm run global-uninstall

This runs npm uninstall -g lilfetch to remove the global npm package and binary.
Followed by rm -rf ~/.lilfetch-venv to delete the user-wide Python venv (including installed deps and Playwright browsers).
Warning: The rm -rf command is irreversible. It only affects the .lilfetch-venv directory in your home folder. Back up if needed (unlikely).

Manual Uninstall (Alternative)

Remove npm package:
```
npm uninstall -g lilfetch
```
Remove Python venv:
```
rm -rf ~/.lilfetch-venv
```
- On Windows: rmdir /s /q %USERPROFILE%\.lilfetch-venv

Local Uninstall

For local installs (e.g., after npm install):

npm uninstall
rm -rf .venv

This removes the local Node modules and repo-specific Python venv.

Verification

npm list -g --depth=0 (no lilfetch).
which lilfetch (empty).
ls ~/.lilfetch-venv (no such file).

For local: rm -rf node_modules .venv and verify no ./.bin/lilfetch.

Development

Edit mcp_server.py for Python logic.
Update bin/lilfetch.js for wrapper changes.
Bump version in package.json, then npm run pack.
For global testing: npm install -g . then npx lilfetch.

Troubleshooting

Permission Errors (Global Install): See Prerequisites for user-owned NPM setup. Avoid sudo—use the config steps.
Python Not Found/Version Error: Ensure Python 3.8+ is in PATH. For pyenv: pyenv install 3.12.0 && pyenv global 3.12.0, then re-run install. Check: python3 --version.
Venv/Deps Fail: For local: Delete .venv and re-run npm install. For global: Delete ~/.lilfetch-venv and re-run npm install -g .. Manual fix (local): python3 -m venv .venv && .venv/bin/pip install -r requirements.txt && .venv/bin/python -m playwright install. Manual fix (global): python3 -m venv ~/.lilfetch-venv && ~/.lilfetch-venv/bin/pip install -r requirements.txt && ~/.lilfetch-venv/bin/python -m playwright install.
Playwright Browsers Missing: Run python -m playwright install in the venv (or manually as logged).
MCP Not Detected in VS Code: Restart VS Code after config; ensure workspace is open correctly.
Uninstall:
- Global: npm uninstall -g lilfetch + rm -rf ~/.lilfetch-venv.
- Local: rm -rf node_modules package-lock.json .venv.

૮₍ • ᴥ • ₎ა

License: MIT