BERDL Datalake MCP Server
A FastAPI-based service that enables AI assistants to interact with Delta Lake tables stored in MinIO through Spark, implementing the Model Context Protocol (MCP) for natural language data operations.
⚠️ Important Warning:
This service allows arbitrary
read-orientedqueries to be executed against Delta Lake tables. Query results will be sent to the model host server, unless you are hosting your model locally.
❌ Additionally, this service is NOT approved for deployment to any production environment, including CI, until explicit approval is granted by KBase leadership. Use strictly for local development or evaluation purposes only.
Documentation
Guides
User Guide - Complete documentation for setting up and using the MCP server:
- Quick Start - Bring the local service up and running
- Creating Sample Delta Tables - Set up local test data
- Using the API - Direct API usage examples
- AI Assistant Integration - Configure and use with MCP Host tools
Claude Code Guide - Using the BERDL MCP server with Claude Code CLI:
- Connect Claude Code to the BERDL data lake
- Query your Delta Lake tables using natural language
- Step-by-step setup and troubleshooting
- Best practices and example queries
Quick Start
Option A: Using Pre-Built Images (Recommended)
-
Clone the repository:
git clone https://github.com/BERDataLakehouse/datalake-mcp-server.git cd datalake-mcp-server -
Edit
docker-compose.yaml:- Uncomment all
image:andplatform:lines - Comment out all
build:sections
- Uncomment all
-
Start the services:
docker compose up -d
Option B: Building from Source (For Developers)
-
Clone required repositories:
# Clone at the same directory level as datalake-mcp-server cd .. git clone https://github.com/BERDataLakehouse/spark_notebook_base.git git clone https://github.com/BERDataLakehouse/kube_spark_manager_image.git git clone https://github.com/BERDataLakehouse/hive_metastore.git cd datalake-mcp-server -
Build base images:
cd ../spark_notebook_base docker build -t spark_notebook_base:local . cd ../datalake-mcp-server -
Ensure
docker-compose.yamlhasbuild:sections uncommented (default) -
Start the services:
docker compose up -d --build
Access the Services
- MCP Server API: http://localhost:8000/apis/mcp/docs
- MCP Server Root: http://localhost:8000/docs
- MinIO Console: http://localhost:9003 (credentials: minio/minio123)
- Spark Master UI: http://localhost:8090
- PostgreSQL: localhost:5432 (credentials: hive/hivepassword)
Note: The MCP server is mounted at /apis/mcp by default. Set SERVICE_ROOT_PATH="" environment variable to serve at root.
Authentication
The service uses KBase authentication with role-based access control:
Environment Variables
| Variable | Required | Default | Description |
|---|---|---|---|
KBASE_AUTH_URL | No | https://ci.kbase.us/services/auth/ | KBase authentication service URL |
KBASE_ADMIN_ROLES | No | KBASE_ADMIN | Comma-separated list of KBase roles with full admin access |
KBASE_REQUIRED_ROLES | No | BERDL_USER | Comma-separated list of KBase roles required to authenticate. Users must have all these roles |
Testing
# Install dependencies (only required on first run or when the uv.lock file changes)
uv sync --locked
# Run tests
PYTHONPATH=. uv run pytest tests
# Run with coverage
PYTHONPATH=. uv run pytest --cov=src tests/