# scrapedatshi — RAG Pipeline API

> scrapedatshi is a pay-as-you-go API for building RAG (Retrieval-Augmented Generation) pipelines.
> Scrape URLs, chunk documents, generate embeddings, and inject into vector databases — all from a single API.

## What scrapedatshi does

scrapedatshi provides a complete RAG data pipeline:

1. **Scrape** — Convert any URL to clean Markdown (strips ads, nav bars, boilerplate)
2. **Chunk** — Split content into RAG-optimized segments (tables and code blocks never split mid-structure)
3. **Embed** — Generate vector embeddings via your own LLM/embedding provider key (BYOK)
4. **Inject** — Upsert vectors directly into your vector database
5. **Extract** — Pull structured data from any URL using your own LLM and a JSON schema

## Key Features

- **AutoRAG** — Crawl an entire site, chunk every page, embed, and inject in one API call. Large sites (>200 pages) are automatically batched server-side.
- **Contextual Retrieval (RAG 2.0)** — Per-chunk LLM context generation, proven to boost retrieval accuracy by 35–50%
- **Schema Extraction** — Extract structured JSON from any URL using your own LLM key
- **JS Rendering** — Headless Chromium support for SPAs and JavaScript-heavy pages
- **BYOK** — Bring your own LLM, embedding, and vector DB keys. scrapedatshi only charges for scraping and orchestration.
- **Claude MCP** — Use all tools directly inside Claude Desktop via the Model Context Protocol

## API Endpoints

- `GET /scrape` — Scrape any URL to clean Markdown
- `POST /v1/rag-chunk` — Scrape + chunk a URL into RAG-ready segments
- `POST /v1/crawl` — Crawl a domain's sitemap and return all page content
- `POST /v1/crawl-chunk` — Crawl + chunk an entire site, return JSON (no VDB)
- `POST /v1/autorag` — Full pipeline: crawl site → chunk → embed → inject into vector DB
- `POST /v1/sync` — Full pipeline for a single URL: scrape → chunk → embed → inject
- `POST /v1/ingest` — Full pipeline for local files: parse → chunk → embed → inject
- `POST /v1/ingest-chunk` — Parse + chunk local files, return JSON (no VDB)
- `POST /v1/extract` — Extract structured data from a URL using your LLM
- `POST /v1/extract-crawl` — Multi-page schema extraction via site crawl
- `POST /api/pdf/text` — Extract text from a PDF (URL or file upload)
- `POST /api/pdf/tables` — Extract tables from a PDF as structured JSON

## Supported Providers

**Embedding:** OpenAI, Cohere, Google Gemini, Mistral, Voyage AI, Ollama (local)

**Vector Databases:** Pinecone, Qdrant, Supabase (pgvector), Weaviate, MongoDB Atlas, Azure Cosmos DB, ChromaDB, LanceDB

**LLM (for extraction + contextual retrieval):** OpenAI, Anthropic, Google Gemini, Cohere, Mistral

## Pricing

Pay-as-you-go credit wallet. No subscription required.

- URL Fetch: $0.0020 / URL
- Spider Fetch: $0.0050 / URL (deep crawl)
- Chunk Fee: $0.0005 / chunk
- Injection Fee: $0.0030 / chunk (vector DB upserts)
- Contextual Retrieval: $0.0010 / enriched chunk
- JS Render: $0.0050 / URL
- Schema Extract: $0.0030 + ($0.0001 × fields) per page

New accounts receive $1.00 free credits. No credit card required.

## Python SDK

Install: `pip install scrapedatshi`

PyPI: https://pypi.org/project/scrapedatshi/
GitHub: https://github.com/scrapedatshi/scrapedatshi-py

```python
from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()  # reads SCRAPEDATSHI_API_KEY from env

# Chunk a URL
result = client.pipeline.chunk_url("https://docs.example.com")
print(f"Got {result.total_chunks} chunks")

# Full AutoRAG pipeline
result = client.pipeline.autorag(
    url="https://docs.example.com",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
)
print(f"Injected {result.vectors_upserted} vectors")
```

## Claude MCP Server

Use all scrapedatshi tools directly inside Claude Desktop — no code required.

Install: `pip install scrapedatshi-mcp`

PyPI: https://pypi.org/project/scrapedatshi-mcp/
GitHub: https://github.com/scrapedatshi/scrapedatshi-mcp

Claude Desktop config (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "scrapedatshi": {
      "command": "uvx",
      "args": ["--refresh", "scrapedatshi-mcp[all]"],
      "env": {
        "SCRAPEDATSHI_API_KEY": "your-api-key"
      }
    }
  }
}
```

Available MCP tools: `scrape_url`, `chunk_file`, `crawl_site`, `extract_data`, `extract_crawl`, `sync_to_vectordb`, `ingest_file`, `autorag`, `verify_provider_key`, `list_embedding_providers`, `list_vector_db_providers`

## Links

- Website: https://scrapedatshi.com
- API Docs: https://scrapedatshi.com/dev
- Sign up: https://scrapedatshi.com/portal/register
- Billing: https://scrapedatshi.com/portal/billing
- Python SDK (PyPI): https://pypi.org/project/scrapedatshi/
- Python SDK (GitHub): https://github.com/scrapedatshi/scrapedatshi-py
- Claude MCP (PyPI): https://pypi.org/project/scrapedatshi-mcp/
- Claude MCP (GitHub): https://github.com/scrapedatshi/scrapedatshi-mcp