CodeRAG
Lightning-fast hybrid code search for AI assistants
Zero dependencies • <50ms search • Hybrid TF-IDF + Vector • MCP ready
Quick Start • Features • MCP Setup • API
Why CodeRAG?
Traditional code search tools are either slow (full-text grep), inaccurate (keyword matching), or complex (require external services).
CodeRAG is different:
❌ Old way: Docker + ChromaDB + Ollama + 30 second startup
✅ CodeRAG: npx @sylphx/coderag-mcp (instant)
| Feature | grep/ripgrep | Cloud RAG | CodeRAG |
|---|---|---|---|
| Semantic understanding | ❌ | ✅ | ✅ |
| Zero external deps | ✅ | ❌ | ✅ |
| Offline support | ✅ | ❌ | ✅ |
| Startup time | Instant | 10-30s | <1s |
| Search latency | ~100ms | ~500ms | <50ms |
✨ Features
Search
- 🔍 Hybrid Search - TF-IDF + optional vector embeddings
- 🧠 StarCoder2 Tokenizer - Code-aware tokenization (4.7MB, trained on code)
- 📊 Smoothed IDF - No term gets ignored, stable ranking
- ⚡ <50ms Latency - Instant results even on large codebases
Indexing
- 🚀 1000-2000 files/sec - Fast initial indexing
- 💾 SQLite Persistence - Instant startup (<100ms) with cached index
- ⚡ Incremental Updates - Smart diff detection, no full rebuilds
- 👁️ File Watching - Real-time index updates on file changes
Integration
- 📦 MCP Server - Works with Claude Desktop, Cursor, VS Code, Windsurf
- 🧠 Vector Search - Optional OpenAI embeddings for semantic search
- 🌳 AST Chunking - Smart code splitting using Synth parsers (15+ languages)
- 💻 Low Memory Mode - SQL-based search for resource-constrained environments
🚀 Quick Start
Option 1: MCP Server (Recommended for AI Assistants)
npx @sylphx/coderag-mcp --root=/path/to/project
Or add to your MCP config:
{
"mcpServers": {
"coderag": {
"command": "npx",
"args": ["-y", "@sylphx/coderag-mcp", "--root=/path/to/project"]
}
}
}
See MCP Server Setup for Claude Desktop, Cursor, VS Code, etc.
Option 2: As a Library
npm install @sylphx/coderag
# or
bun add @sylphx/coderag
import { CodebaseIndexer, PersistentStorage } from '@sylphx/coderag'
// Create indexer with persistent storage
const storage = new PersistentStorage({ codebaseRoot: './my-project' })
const indexer = new CodebaseIndexer({
codebaseRoot: './my-project',
storage,
})
// Index codebase (instant on subsequent runs)
await indexer.index({ watch: true })
// Search
const results = await indexer.search('authentication logic', { limit: 10 })
console.log(results)
// [{ path: 'src/auth/login.ts', score: 0.87, matchedTerms: ['authentication', 'logic'], snippet: '...' }]
📦 Packages
| Package | Description | Install |
|---|---|---|
| @sylphx/coderag | Core search library | npm i @sylphx/coderag |
| @sylphx/coderag-mcp | MCP server for AI assistants | npx @sylphx/coderag-mcp |
🔌 MCP Server Setup
Claude Desktop
Add to claude_desktop_config.json:
macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
{
"mcpServers": {
"coderag": {
"command": "npx",
"args": ["-y", "@sylphx/coderag-mcp", "--root=/path/to/project"]
}
}
}
Cursor
Add to ~/.cursor/mcp.json (macOS) or %USERPROFILE%\.cursor\mcp.json (Windows):
{
"mcpServers": {
"coderag": {
"command": "npx",
"args": ["-y", "@sylphx/coderag-mcp", "--root=/path/to/project"]
}
}
}
VS Code
Add to VS Code settings (JSON) or .vscode/mcp.json:
{
"mcp": {
"servers": {
"coderag": {
"command": "npx",
"args": ["-y", "@sylphx/coderag-mcp", "--root=${workspaceFolder}"]
}
}
}
}
Windsurf
Add to ~/.codeium/windsurf/mcp_config.json:
{
"mcpServers": {
"coderag": {
"command": "npx",
"args": ["-y", "@sylphx/coderag-mcp", "--root=/path/to/project"]
}
}
}
Claude Code
claude mcp add coderag -- npx -y @sylphx/coderag-mcp --root=/path/to/project
🛠️ MCP Tool: codebase_search
Search project source files with hybrid TF-IDF + vector ranking.
Parameters
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
query | string | Yes | - | Search query |
limit | number | No | 10 | Max results |
include_content | boolean | No | true | Include code snippets |
file_extensions | string[] | No | - | Filter by extension (e.g., [".ts", ".tsx"]) |
path_filter | string | No | - | Filter by path pattern |
exclude_paths | string[] | No | - | Exclude paths (e.g., ["node_modules", "dist"]) |
Example
{
"query": "user authentication login",
"limit": 5,
"file_extensions": [".ts", ".tsx"],
"exclude_paths": ["node_modules", "dist", "test"]
}
Response Format
LLM-optimized output (minimal tokens, maximum content):
# Search: "user authentication login" (3 results)
## src/auth/login.ts:15-28
```typescript
15: export async function authenticate(credentials) {
16: const user = await findUser(credentials.email)
17: return validatePassword(user, credentials.password)
18: }
src/middleware/auth.ts:42-55 [md→typescript]
42: // Embedded code from markdown docs
43: const authMiddleware = (req, res, next) => {
src/utils/large.ts:1-200 [truncated]
1: // First 70% shown...
... [800 chars truncated] ...
195: // Last 20% shown
---
## 📚 API Reference
### `CodebaseIndexer`
Main class for indexing and searching.
```typescript
import { CodebaseIndexer, PersistentStorage } from '@sylphx/coderag'
const storage = new PersistentStorage({ codebaseRoot: './project' })
const indexer = new CodebaseIndexer({
codebaseRoot: './project',
storage,
maxFileSize: 1024 * 1024, // 1MB default
})
// Index with file watching
await indexer.index({ watch: true })
// Search with options
const results = await indexer.search('query', {
limit: 10,
includeContent: true,
fileExtensions: ['.ts', '.js'],
excludePaths: ['node_modules'],
})
// Stop watching
await indexer.stopWatch()
PersistentStorage
SQLite-backed storage for instant startup.
import { PersistentStorage } from '@sylphx/coderag'
const storage = new PersistentStorage({
codebaseRoot: './project', // Creates .coderag/ folder
dbPath: './custom.db', // Optional custom path
})
Low-Level TF-IDF Functions
import { buildSearchIndex, searchDocuments, initializeTokenizer } from '@sylphx/coderag'
// Initialize StarCoder2 tokenizer (4.7MB, one-time download)
await initializeTokenizer()
// Build index
const documents = [
{ uri: 'file://auth.ts', content: 'export function authenticate...' },
{ uri: 'file://user.ts', content: 'export class User...' },
]
const index = await buildSearchIndex(documents)
// Search
const results = await searchDocuments('authenticate user', index, { limit: 5 })
Vector Search (Optional)
For semantic search with embeddings:
import { hybridSearch, createEmbeddingProvider } from '@sylphx/coderag'
// Requires OPENAI_API_KEY environment variable
const results = await hybridSearch('authentication flow', indexer, {
vectorWeight: 0.7, // 70% vector, 30% TF-IDF
limit: 10,
})
⚙️ Configuration
MCP Server Options
| Option | Default | Description |
|---|---|---|
--root=<path> | Current directory | Codebase root path |
--max-size=<bytes> | 1048576 (1MB) | Max file size to index |
--no-auto-index | false | Disable auto-indexing on startup |
Environment Variables
| Variable | Description |
|---|---|
OPENAI_API_KEY | Enable vector search with OpenAI embeddings |
OPENAI_BASE_URL | Custom OpenAI-compatible endpoint |
EMBEDDING_MODEL | Embedding model (default: text-embedding-3-small) |
EMBEDDING_DIMENSIONS | Custom embedding dimensions |
📊 Performance
| Metric | Value |
|---|---|
| Initial indexing | ~1000-2000 files/sec |
| Startup with cache | <100ms |
| Search latency | <50ms |
| Memory per 1000 files | ~1-2 MB |
| Tokenizer size | 4.7MB (StarCoder2) |
Benchmarks
Tested on MacBook Pro M1, 16GB RAM:
| Codebase | Files | Index Time | Search Time |
|---|---|---|---|
| Small (100 files) | 100 | 0.5s | <10ms |
| Medium (1000 files) | 1,000 | 2s | <30ms |
| Large (10000 files) | 10,000 | 15s | <50ms |
🏗️ Architecture
coderag/
├── packages/
│ ├── core/ # @sylphx/coderag
│ │ ├── src/
│ │ │ ├── indexer.ts # Main indexer with file watching
│ │ │ ├── tfidf.ts # TF-IDF with StarCoder2 tokenizer
│ │ │ ├── code-tokenizer.ts # StarCoder2 tokenization
│ │ │ ├── hybrid-search.ts # Vector + TF-IDF fusion
│ │ │ ├── incremental-tfidf.ts # Smart incremental updates
│ │ │ ├── storage-persistent.ts # SQLite storage
│ │ │ ├── vector-storage.ts # LanceDB vector storage
│ │ │ ├── embeddings.ts # OpenAI embeddings
│ │ │ ├── ast-chunking.ts # Synth AST chunking
│ │ │ └── language-config.ts # Language registry (15+ languages)
│ │ └── package.json
│ │
│ └── mcp-server/ # @sylphx/coderag-mcp
│ ├── src/
│ │ └── index.ts # MCP server
│ └── package.json
How It Works
- Indexing: Scans codebase, tokenizes with StarCoder2, builds TF-IDF index
- AST Chunking: Splits code at semantic boundaries (functions, classes, etc.)
- Storage: Persists to SQLite (
.coderag/folder) for instant startup - Watching: Detects file changes, performs incremental updates
- Search: Hybrid TF-IDF + optional vector search with score fusion
Supported Languages
AST-based chunking with semantic boundary detection:
| Category | Languages |
|---|---|
| JavaScript | JavaScript, TypeScript, JSX, TSX |
| Systems | Python, Go, Java, C |
| Markup | Markdown, HTML, XML |
| Data/Config | JSON, YAML, TOML, INI |
| Other | Protobuf |
Embedded Code Support: Automatically parses code blocks in Markdown and <script>/<style> tags in HTML.
🔧 Development
# Clone
git clone https://github.com/SylphxAI/coderag.git
cd coderag
# Install
bun install
# Build
bun run build
# Test
bun run test
# Lint & Format
bun run lint
bun run format
🤝 Contributing
Contributions are welcome! Please:
- Open an issue to discuss changes
- Fork and create a feature branch
- Run
bun run lintandbun run test - Submit a pull request
📄 License
MIT © Sylphx
Powered by Sylphx
Built with @sylphx/synth • @sylphx/mcp-server-sdk • @sylphx/doctor • @sylphx/bump