scrapedatshi icon scrapedatshi
🚀 Get API Key Sign in →

One API key unlocks all tools — scraping, PDF extraction, RAG chunking, vector injection, schema extraction, and more. No credit card required. New accounts receive $1.00 free credits.

🐍 Python SDK pip install scrapedatshi Typed models · Sync + Async · IDE autocomplete
PyPI → GitHub →
🤖 Claude MCP pip install scrapedatshi-mcp Use all tools directly inside Claude Desktop
PyPI → Setup →

API Reference

All endpoints require X-API-Key: YOUR_KEY in the request header. The Python SDK handles authentication automatically.

POST/v1/rag-chunk
client.pipeline.chunk_url()

Scrape any URL, convert to clean Markdown, and split into RAG-optimized chunks. Tables and code blocks are never split mid-structure. PDF URLs are automatically detected and extracted.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()  # reads SCRAPEDATSHI_API_KEY from env

# Basic scrape + chunk
result = client.pipeline.chunk_url("https://docs.example.com")
print(f"Got {result.total_chunks} chunks — cost ${result.credits_used:.4f}")
for chunk in result.chunks:
    print(chunk.content[:80])

# With CSS selector to target main content
result = client.pipeline.chunk_url(
    "https://docs.example.com/guide",
    selector="article",
    chunk_size=512,
    overlap=50,
)

# With Contextual Retrieval (RAG 2.0) — boosts retrieval accuracy 35–50%
result = client.pipeline.chunk_url(
    "https://docs.example.com",
    contextual_retrieval=True,
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
)
for chunk in result.chunks:
    print(chunk.context)        # per-chunk LLM context
    print(chunk.original_text)  # raw text before enrichment

# JS rendering for SPAs and JavaScript-heavy pages
result = client.pipeline.chunk_url(
    "https://spa.example.com",
    js_render=True,
)
Billing: $0.0020 / URL (local fetch, default) · $0.0040 / URL (server fetch) · $0.0005 / chunk · +$0.0010 / chunk (contextual retrieval)
POST/v1/process-text
client.pipeline.chunk_file()

Parse a local file and chunk its content into RAG-ready segments. Supports PDF, MD, TXT, YAML, and JSON. In local-fetch mode (default), the file is parsed on your machine — no heavy PDF processing on our server.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Chunk a local PDF (parsed on your machine, text sent to server for chunking)
result = client.pipeline.chunk_file("./docs/manual.pdf")
print(f"Got {result.total_chunks} chunks from {result.source}")
print(f"Cost: ${result.credits_used:.4f}")

# Also supports: .md, .txt, .yaml, .yml, .json
result = client.pipeline.chunk_file("./README.md", chunk_size=256)

# For scanned/image-only PDFs that need OCR — use server mode
from scrapedatshi import ScrapedatshiClient
client = ScrapedatshiClient(fetch_mode="server")
result = client.pipeline.chunk_file("./scanned_report.pdf")  # OCR included
Billing: $0.0020 / file (local parse, default) · $0.0040 / file (server parse + OCR) · $0.0005 / chunk
POST/v1/crawl-chunk
client.pipeline.crawl()

Crawl an entire website and chunk all pages. Two modes: sitemap (reads sitemap.xml — best for docs/blogs) and spider (follows links — works on any site). Large sites are automatically batched server-side.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Sitemap crawl (default) — reads sitemap.xml
result = client.pipeline.crawl("https://docs.example.com", max_pages=20)
print(f"Crawled {result.pages_crawled} pages → {result.total_chunks} chunks")
print(f"Cost: ${result.credits_used:.4f}")

# Spider crawl — follows links, works on any site
result = client.pipeline.crawl(
    "https://example.com",
    crawl_mode="spider",
    max_pages=10,
    include_pattern="/docs/",
    exclude_pattern="/blog/",
)

# Authenticated crawl — cookies stay on your machine (v0.10.0+)
result = client.pipeline.crawl(
    "https://internal.company.com",
    cookies={"session": "abc123"},
    headers={"Authorization": "Bearer eyJ..."},
    max_pages=20,
)

# Subdomain scope — also crawls wiki.company.com, docs.company.com
result = client.pipeline.crawl(
    "https://company.com",
    cookies={"session": "abc123"},
    allow_subdomains=True,
    max_pages=30,
)
Billing: $0.0020 / URL (local fetch, default) · $0.0050 / URL (spider, server) · $0.0005 / chunk
POST/v1/sync
client.pipeline.sync()

Full pipeline for a single URL: scrape → chunk → embed → inject into your vector database. You bring your own embedding provider and vector DB keys (BYOK).

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.sync(
    url="https://docs.example.com",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={
        "api_key": "pc-...",
        "index_host": "https://my-index-abc123.svc.pinecone.io",
    },
)
print(f"Upserted {result.vectors_upserted} vectors ({result.total_tokens} tokens)")
print(f"Cost: ${result.credits_used:.4f}")

# Qdrant example
result = client.pipeline.sync(
    url="https://docs.example.com",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="qdrant",
    vector_db_config={
        "url": "https://your-cluster.qdrant.io",
        "collection_name": "docs",
        "api_key": "qdrant-key",
    },
)
Billing: $0.0020 / URL · $0.0005 / chunk · $0.0030 / chunk injected
POST/v1/ingest
client.pipeline.ingest()

Full pipeline for local files: parse → chunk → embed → inject into your vector database. Supports PDF, MD, TXT, YAML, and JSON.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.ingest(
    file_path="./docs/manual.pdf",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={
        "api_key": "pc-...",
        "index_host": "https://my-index-abc123.svc.pinecone.io",
    },
)
print(f"Ingested {result.chunks_created} chunks → {result.vectors_upserted} vectors")
print(f"Cost: ${result.credits_used:.4f}")
Billing: $0.0020 / file · $0.0005 / chunk · $0.0030 / chunk injected
POST/v1/autorag
client.pipeline.autorag()

Full AutoRAG pipeline in one call: crawl an entire domain → chunk every page → embed → inject into your vector database. Large sites (>200 pages) are automatically batched server-side.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.autorag(
    url="https://docs.example.com",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={
        "api_key": "pc-...",
        "index_host": "https://my-index-abc123.svc.pinecone.io",
    },
    crawl_mode="sitemap",   # or "spider"
    max_pages=50,
    include_pattern="/docs/",
)
print(f"Crawled {result.pages_crawled} pages → {result.vectors_upserted} vectors")
print(f"Cost: ${result.credits_used:.4f}")

# Large sites are auto-batched — no manual pagination needed
result = client.pipeline.autorag(
    url="https://large-docs-site.com",
    max_pages=500,  # processed as multiple batches of 200
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
)
Billing: $0.0020 / URL · $0.0005 / chunk · $0.0030 / chunk injected
POST/v1/extract
client.pipeline.extract()

Extract structured JSON from any URL using your own LLM key. Define a schema and the API returns a typed JSON object — or a list of objects for pages with multiple items.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Extract a single object
result = client.pipeline.extract(
    url="https://example.com/products/widget-pro",
    schema={
        "title": "string — the product name",
        "price": "number — the price in USD",
        "in_stock": "boolean — whether the item is in stock",
    },
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
)
print(result.extracted)
# → {"title": "Widget Pro", "price": 29.99, "in_stock": True}
print(f"Cost: ${result.credits_used:.4f}")

# Extract a list of items from a listing page
result = client.pipeline.extract(
    url="https://example.com/products",
    schema={
        "title": "string — the product name",
        "price": "number — the price in USD",
    },
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
    extract_as_list=True,
)
print(f"Extracted {result.item_count} products")
Billing: $0.0020 + $0.0030 + ($0.0001 × N fields) per page
POST/v1/extract-crawl
client.pipeline.extract_crawl()

Crawl an entire domain and extract structured data from every page using your LLM. Each page is processed independently — failed pages return an error without aborting the batch. Only successfully extracted pages are billed.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.extract_crawl(
    url="https://example.com/products",
    schema={
        "title": "string — the product name",
        "price": "number — the price in USD",
        "in_stock": "boolean — whether the item is in stock",
    },
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
    crawl_mode="sitemap",   # or "spider"
    max_pages=20,
    include_pattern="/products/",
)
print(f"Extracted {result.pages_extracted}/{result.pages_attempted} pages")
print(f"Cost: ${result.credits_used:.4f}")

# Iterate results
for page in result.results:
    if page.ok:
        print(f"  {page.url}: {page.extracted}")
    else:
        print(f"  {page.url}: FAILED — {page.error}")

# Access only successful results
for page in result.successful_results:
    print(page.extracted["title"], page.extracted["price"])
Billing: $0.0020 + $0.0030 + ($0.0001 × N fields) per successfully extracted page only
POST/v1/query
client.pipeline.query_vectordb()

Semantic search: embed a natural language query and retrieve the most relevant chunks from your vector database. Use inspect_vectordb() first to confirm the correct embedding model.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Step 1: Inspect to confirm the embedding model used during ingestion
inspect = client.pipeline.inspect_vectordb(
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
)
print(f"Dimension: {inspect.dimension}")
print(f"Suggested models: {[m.label for m in inspect.suggested_models]}")

# Step 2: Query with the confirmed model
result = client.pipeline.query_vectordb(
    query="How do I authenticate with the API?",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",  # must match ingestion model
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
    top_k=5,
)
print(f"Found {result.chunks_retrieved} results (cost: ${result.credits_used:.4f})")
for r in result.results:
    print(f"  [{r.score:.2f}] {r.text[:100]}...")
Billing: $0.0002 / chunk retrieved · inspect_vectordb() is always free
POST/v1/rag-chat
client.pipeline.rag_chat()

RAG Chat: embed a query, retrieve the most relevant chunks from your vector database, and generate a grounded LLM answer. You bring your own LLM key — scrapedatshi only charges for the vector retrieval.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.rag_chat(
    query="How do I authenticate with the API?",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",  # must match ingestion model
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
    top_k=5,
)
print(result.answer)
print(f"Based on {result.chunks_retrieved} chunks (cost: ${result.credits_used:.4f})")
for source in result.sources:
    print(f"  [{source.score:.2f}] {source.text[:80]}...")
Billing: $0.0002 / chunk retrieved · LLM tokens are your own cost (not billed by scrapedatshi)
POST/v1/inspect-vectordb
client.pipeline.inspect_vectordb()

Read vector database metadata — dimension, vector count, and suggested embedding models based on the detected dimension. Use this before querying to confirm which embedding model was used during ingestion. Always free — no credits charged.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.inspect_vectordb(
    vector_db="pinecone",
    vector_db_config={
        "api_key": "pc-...",
        "index_host": "https://my-index-abc123.svc.pinecone.io",
    },
)
print(f"Dimension: {result.dimension}")
print(f"Total vectors: {result.total_vector_count:,}")
print(f"Suggested models:")
for model in result.suggested_models:
    print(f"  • {model.label} ({model.provider} / {model.model})")
if result.note:
    print(f"Note: {result.note}")
Billing: Free — no credits deducted
New in v0.10.0
cookies= · headers= · allow_subdomains=

For pages behind a login wall, pass your session cookies and/or custom headers to any fetch method. Credentials are used exclusively on your machine and never transmitted to our servers. The credential shield ensures cookies are only sent to URLs within the permitted domain scope — never leaked to external domains discovered during crawling.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Scrape a login-walled page
result = client.pipeline.chunk_url(
    "https://internal.company.com/wiki/api-docs",
    cookies={"session": "abc123", "csrf": "xyz"},
    headers={"Authorization": "Bearer eyJ..."},
)

# Authenticated sitemap crawl
result = client.pipeline.crawl(
    "https://internal.company.com",
    cookies={"session": "abc123"},
    headers={"Authorization": "Bearer eyJ..."},
    max_pages=20,
)

# Spider crawl with subdomain scope
# Crawls company.com + wiki.company.com + docs.company.com
# Cookies only sent to URLs within the same apex domain
result = client.pipeline.crawl(
    "https://company.com",
    crawl_mode="spider",
    cookies={"session": "abc123"},
    allow_subdomains=True,   # multi-part TLDs (.co.uk) handled safely
    max_pages=30,
)

# Full pipeline with authentication
result = client.pipeline.sync(
    url="https://internal.company.com/wiki/api-docs",
    cookies={"session": "abc123"},
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
)
Security: Cookies and headers are processed locally on your machine. They are never transmitted to scrapedatshi servers. You are responsible for ensuring your use of session credentials complies with the terms of service of any third-party website you access.
pip install scrapedatshi-mcp

Use all scrapedatshi tools directly inside Claude Desktop — no code required. Just talk to Claude naturally: "Crawl https://docs.example.com and sync it to my Pinecone index using OpenAI embeddings."

Setup

Open Claude Desktop → Settings → Developer → Edit Config, then add: 

{
  "mcpServers": {
    "scrapedatshi": {
      "command": "uvx",
      "args": [
        "--from", "scrapedatshi-mcp[all]",
        "--refresh",
        "scrapedatshi-mcp"
      ],
      "env": {
        "SCRAPEDATSHI_API_KEY": "sds_your_key_here",
        "OPENAI_API_KEY": "sk-...",
        "PINECONE_API_KEY": "pc-..."
      }
    }
  }
}

[all] installs all provider SDKs. --refresh auto-updates on every Claude Desktop restart. Restart Claude Desktop after saving.

Available MCP Tools

scrape_url

Scrape & chunk a URL

chunk_file

Chunk a local file

crawl_site

Crawl an entire site

extract_data

Extract structured fields

extract_crawl

Multi-page extraction

sync_to_vectordb

URL → embed → inject

ingest_file

File → embed → inject

autorag

Crawl site → embed → inject

inspect_vectordb

Read VDB metadata (free)

query_vectordb

Semantic search

rag_chat

Retrieve + generate answer

verify_provider_key

Verify API key + get models

list_embedding_providers

List embedding providers

list_vector_db_providers

List vector DB providers

get_usage_guide

Tool selection guide

Example conversations

💬 "Scrape https://docs.example.com and give me the chunks"
💬 "Crawl https://example.com/products and extract the title and price from every page"
💬 "Sync https://docs.example.com to my Pinecone index using OpenAI text-embedding-3-small"
💬 "Crawl the entire docs.stripe.com site and inject it into my Pinecone index"
💬 "Query my Pinecone index for information about API authentication"
💬 "Scrape https://internal.company.com/wiki — use my session cookie: abc123"
PyPI → GitHub →

Supported Providers

Embedding

OpenAI · Cohere · Google Gemini · Mistral · Voyage AI · Ollama (local)

Vector Databases

Pinecone · Qdrant · Supabase (pgvector) · Weaviate · MongoDB Atlas · Azure Cosmos DB · ChromaDB · LanceDB

LLM (extraction + CR)

OpenAI · Anthropic · Google Gemini

Use from scrapedatshi.providers import EMBEDDING_PROVIDERS, VECTOR_DB_PROVIDERS, LLM_PROVIDERS to discover all supported providers and required config fields programmatically.