scrapedatshi icon scrapedatshi
🚀 Get API Key Sign in →

One API key unlocks all tools — scraping, PDF extraction, RAG chunking, vector injection, schema extraction, and more.

🐍 Python SDK pip install scrapedatshi Typed models · Sync + Async · IDE autocomplete · scrapedatshi init scaffold
PyPI → GitHub →
🤖 Claude MCP pip install scrapedatshi-mcp Use all tools directly inside Claude Desktop
PyPI → Setup →

API Reference

All endpoints require X-API-Key: YOUR_KEY in the request header. The Python SDK handles authentication automatically.

POST/v1/process-html
client.pipeline.scrape_url() client.pipeline.scrape_file()

Scrape a URL and return the full page content as clean Markdown — no chunking, no embedding, just the raw text in one piece. Use this when you want to read, summarize, or display a page's content directly. For RAG pipelines that need the content split into segments, use Chunk URL instead.

scrape_file() does the same for local files — extracts text from PDF, MD, TXT, DOCX, etc. and returns it as a single Markdown string. 

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()  # reads SCRAPEDATSHI_API_KEY from env

# Optional: uncomment to target a specific section
# SELECTOR = "article"   # detected sections are printed below — copy one here

# Scrape a URL → get clean Markdown (no chunking)
result = client.pipeline.scrape_url(
    "https://docs.example.com/guide",
    # selector=SELECTOR,   # uncomment after choosing a section below
)
print(result.markdown)          # full page as Markdown
print(result.title)             # page title (if detected)
print(f"Cost: ${result.credits_used:.4f}")
# Detected content sections — uncomment SELECTOR above and re-run to target one:
if result.selectors_found:
    print(f"Sections: {result.selectors_found}")

# Scrape a local file → get Markdown (PDF, MD, TXT, DOCX, etc.)
result = client.pipeline.scrape_file("./docs/manual.pdf")
print(result.markdown)
print(f"Cost: ${result.credits_used:.4f}")
Billing: $0.0020 / URL (local fetch, default — recommended) · $0.0040 / URL (server fetch) · $0.0020 / file (scrape_file, direct parse)
POST/v1/rag-chunk
client.pipeline.chunk_url()

Scrape any URL, convert to clean Markdown, and split into RAG-optimized chunks. Tables and code blocks are never split mid-structure. PDF URLs are automatically detected and extracted.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()  # reads SCRAPEDATSHI_API_KEY from env

# Basic scrape + chunk
result = client.pipeline.chunk_url("https://docs.example.com")
print(f"Got {result.total_chunks} chunks — cost ${result.credits_used:.4f}")
for chunk in result.chunks:
    print(chunk.content[:80])

# With CSS selector to target main content
result = client.pipeline.chunk_url(
    "https://docs.example.com/guide",
    selector="article",
    chunk_size=512,
    overlap=50,
)

# Hierarchical (parent-child) chunking — small child chunks for embedding,
# large parent chunks stored as metadata for LLM context on retrieval
result = client.pipeline.chunk_url(
    "https://docs.example.com",
    hierarchical=True,        # enable parent-child chunking
    child_chunk_size=128,     # child chunk size (default: 128 tokens)
    chunk_size=512,           # parent chunk size (default: 512 tokens)
)
for chunk in result.chunks:
    print(f"Child ({chunk.token_estimate} tokens): {chunk.content[:80]}")
    if chunk.parent_text:
        print(f"Parent: {chunk.parent_text[:120]}")  # feed this to the LLM

# With Contextual Retrieval (RAG 2.0) — boosts retrieval accuracy 35–50%
result = client.pipeline.chunk_url(
    "https://docs.example.com",
    contextual_retrieval=True,
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
)
for chunk in result.chunks:
    print(chunk.context)        # per-chunk LLM context
    print(chunk.original_text)  # raw text before enrichment

# JS rendering for SPAs and JavaScript-heavy pages
result = client.pipeline.chunk_url(
    "https://spa.example.com",
    js_render=True,
)
Billing: $0.0020 / URL (local fetch, default — recommended) · $0.0040 / URL (server fetch) · $0.0005 / chunk · +$0.0010 / chunk (contextual retrieval)
POST/v1/process-text
client.pipeline.chunk_file()

Parse a local file and chunk its content into RAG-ready segments. Supports PDF, MD, TXT, YAML, JSON, CSV, XLSX, DOCX, IPYNB, HTML, XML, and all common code files (.py, .js, .ts, .sql, .go, .rb, .java, etc.). In local-fetch mode (default), the file is parsed on your machine — no heavy PDF processing on our server.

Code-aware chunking: .py files are split by top-level class and function using Python's AST parser — each class/function becomes its own chunk with imports prepended for context. .sql files are split by statement block (CREATE TABLE, INSERT INTO, SELECT, etc.). Both fall back to plain text if parsing fails. 

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Chunk a local PDF (parsed on your machine, text sent to server for chunking)
result = client.pipeline.chunk_file("./docs/manual.pdf")
print(f"Got {result.total_chunks} chunks from {result.source}")
print(f"Cost: ${result.credits_used:.4f}")

# Python files — AST-aware: each class/function becomes its own chunk
result = client.pipeline.chunk_file("./src/models.py")
# → Each class and top-level function is a separate chunk
# → Imports are prepended to each unit for context

# SQL files — statement-aware: each CREATE TABLE / SELECT / etc. is its own chunk
result = client.pipeline.chunk_file("./schema.sql")

# For scanned/image-only PDFs that need OCR — use server mode
client = ScrapedatshiClient(fetch_mode="server")
result = client.pipeline.chunk_file("./scanned_report.pdf")  # OCR included
Billing: $0.0020 / file (local parse, default) · $0.0040 / file (server parse + OCR) · $0.0005 / chunk
POST/v1/crawl-chunk
client.pipeline.crawl()

Crawl an entire website and chunk all pages. Two modes: sitemap (reads sitemap.xml — best for docs/blogs) and spider (follows links — works on any site). Direct-fetch mode (default) gives you maximum throughput — requests run straight from your pipeline without queuing or shared infrastructure.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Sitemap crawl (default) — reads sitemap.xml
result = client.pipeline.crawl("https://docs.example.com", max_pages=20)
print(f"Crawled {result.pages_crawled} pages → {result.total_chunks} chunks")
print(f"Cost: ${result.credits_used:.4f}")

# Spider crawl — follows links, works on any site
result = client.pipeline.crawl(
    "https://example.com",
    crawl_mode="spider",
    max_pages=10,
    include_pattern="/docs/",
    exclude_pattern="/blog/",
)

# Authenticated crawl — cookies stay on your machine (v0.10.0+)
result = client.pipeline.crawl(
    "https://internal.company.com",
    cookies={"session": "abc123"},
    headers={"Authorization": "Bearer eyJ..."},
    max_pages=20,
)

# Subdomain scope — also crawls wiki.company.com, docs.company.com
result = client.pipeline.crawl(
    "https://company.com",
    cookies={"session": "abc123"},
    allow_subdomains=True,
    max_pages=30,
)
Billing: $0.0020 / URL (local fetch, default — recommended) · $0.0050 / URL (spider, server fetch) · $0.0005 / chunk
POST/v1/sync
client.pipeline.sync()

Full pipeline for a single URL: scrape → chunk → embed → inject into your vector database. You bring your own embedding provider and vector DB keys (BYOK). Supports hierarchical (parent-child) chunking — small child chunks are embedded for precise vector matching; the larger parent chunk is stored as metadata for LLM context on retrieval.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.sync(
    url="https://docs.example.com",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={
        "api_key": "pc-...",
        "index_host": "https://my-index-abc123.svc.pinecone.io",
    },
    # hierarchical=True,      # optional: parent-child chunking for better retrieval
    # child_chunk_size=128,   # child chunk size when hierarchical=True (default: 128)
)
print(f"Upserted {result.vectors_upserted} vectors ({result.total_tokens} tokens)")
print(f"Cost: ${result.credits_used:.4f}")

# Qdrant example with hierarchical chunking
result = client.pipeline.sync(
    url="https://docs.example.com",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="qdrant",
    vector_db_config={
        "url": "https://your-cluster.qdrant.io",
        "collection_name": "docs",
        "api_key": "qdrant-key",
    },
    hierarchical=True,   # child chunks embedded, parent_text stored in metadata
)
Billing: $0.0020 / URL · $0.0005 / chunk · $0.0030 / chunk injected
POST/v1/ingest
client.pipeline.ingest()

Full pipeline for local files: parse → chunk → embed → inject into your vector database. Supports PDF, MD, TXT, YAML, JSON, CSV, XLSX, DOCX, IPYNB, HTML, XML, and all common code files. Supports hierarchical (parent-child) chunking — small child chunks are embedded for precise vector matching; the larger parent chunk is stored as metadata for LLM context on retrieval.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.ingest(
    file_path="./docs/manual.pdf",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={
        "api_key": "pc-...",
        "index_host": "https://my-index-abc123.svc.pinecone.io",
    },
    # hierarchical=True,      # optional: parent-child chunking for better retrieval
    # child_chunk_size=128,   # child chunk size when hierarchical=True (default: 128)
)
print(f"Ingested {result.chunks_created} chunks → {result.vectors_upserted} vectors")
print(f"Cost: ${result.credits_used:.4f}")
Billing: $0.0020 / file · $0.0005 / chunk · $0.0030 / chunk injected
client.pipeline.ingest_scraped()

Bulk-ingest a folder of pre-scraped files into your vector database. Designed for output from web scrapers (Scrapy, Playwright, Apify, custom scripts). Supports all common file types including .md, .txt, .json, .yaml, .yml, .csv, .xlsx, .docx, .py, .sql, and more. JSON arrays are automatically detected and each item is extracted and ingested individually. .py files are split by class/function (AST-aware); .sql files by statement block. Includes exponential backoff on rate limits and configurable per-file delay. 

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Ingest all supported files in a folder
result = client.pipeline.ingest_scraped(
    folder_path="./scrapy_output/",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={
        "api_key": "pc-...",
        "index_host": "https://my-index-abc123.svc.pinecone.io",
    },
)
print(f"Processed {result.files_processed} files → {result.vectors_upserted} vectors")
print(f"Failed: {result.files_failed} files")
print(f"Cost: ${result.credits_used:.4f}")
for err in result.errors:
    print(f"  ✗ {err['file']} — {err['error']}")

# Scrapy JSON dump (array of items) — each item ingested individually
# Scrapy items with 'text', 'content', 'html', 'body', 'markdown', or 'description'
# fields are automatically detected and extracted
result = client.pipeline.ingest_scraped(
    folder_path="./",
    file_extensions=[".json"],   # only process JSON files
    batch_delay=1.0,             # 1s pause between files (rate limit safety)
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
)

# Async version
result = await client.pipeline.ingest_scraped_async(
    folder_path="./docs/",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="qdrant",
    vector_db_config={
        "url": "https://your-cluster.qdrant.io",
        "collection_name": "docs",
        "api_key": "qdrant-key",
    },
)
Billing: $0.0020 / file · $0.0005 / chunk · $0.0030 / chunk injected · Only successfully processed files are billed
POST/v1/autorag
client.pipeline.autorag()

Full AutoRAG pipeline in one call: crawl an entire domain → chunk every page → embed → inject into your vector database. Direct-fetch mode (default) gives you maximum throughput — requests run straight from your pipeline for fast, unqueued processing at scale.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.autorag(
    url="https://docs.example.com",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={
        "api_key": "pc-...",
        "index_host": "https://my-index-abc123.svc.pinecone.io",
    },
    crawl_mode="sitemap",   # or "spider"
    max_pages=50,
    include_pattern="/docs/",
)
print(f"Crawled {result.pages_crawled} pages → {result.vectors_upserted} vectors")
print(f"Cost: ${result.credits_used:.4f}")

# Scale to large sites — set max_pages as high as you need
result = client.pipeline.autorag(
    url="https://large-docs-site.com",
    max_pages=500,
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
)
Billing: $0.0020 / URL · $0.0005 / chunk · $0.0030 / chunk injected
POST/v1/extract
client.pipeline.extract()

Extract structured JSON from any URL using your own LLM key. Define a schema and the API returns a typed JSON object — or a list of objects for pages with multiple items.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Extract a single object
result = client.pipeline.extract(
    url="https://example.com/products/widget-pro",
    schema={
        "title": "string — the product name",
        "price": "number — the price in USD",
        "in_stock": "boolean — whether the item is in stock",
    },
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
)
print(result.extracted)
# → {"title": "Widget Pro", "price": 29.99, "in_stock": True}
print(f"Cost: ${result.credits_used:.4f}")

# Extract a list of items from a listing page
result = client.pipeline.extract(
    url="https://example.com/products",
    schema={
        "title": "string — the product name",
        "price": "number — the price in USD",
    },
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
    extract_as_list=True,
)
print(f"Extracted {result.item_count} products")
Billing: $0.0020 + $0.0030 + ($0.0001 × N fields) per page
POST/v1/extract-crawl
client.pipeline.extract_crawl()

Crawl an entire domain and extract structured data from every page using your LLM. Each page is processed independently — failed pages return an error without aborting the batch. Only successfully extracted pages are billed.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.extract_crawl(
    url="https://example.com/products",
    schema={
        "title": "string — the product name",
        "price": "number — the price in USD",
        "in_stock": "boolean — whether the item is in stock",
    },
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
    crawl_mode="sitemap",   # or "spider"
    max_pages=20,
    include_pattern="/products/",
)
print(f"Extracted {result.pages_extracted}/{result.pages_attempted} pages")
print(f"Cost: ${result.credits_used:.4f}")

# Iterate results
for page in result.results:
    if page.ok:
        print(f"  {page.url}: {page.extracted}")
    else:
        print(f"  {page.url}: FAILED — {page.error}")

# Access only successful results
for page in result.successful_results:
    print(page.extracted["title"], page.extracted["price"])
Billing: $0.0020 + $0.0030 + ($0.0001 × N fields) per successfully extracted page only
POST/v1/query
client.pipeline.query_vectordb()

Semantic search: embed a natural language query and retrieve the most relevant chunks from your vector database. Supports hybrid search (vector + BM25 keyword + Reciprocal Rank Fusion) for improved cross-referencing accuracy. Use inspect_vectordb() first to confirm the correct embedding model.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Step 1: Inspect to confirm the embedding model used during ingestion
inspect = client.pipeline.inspect_vectordb(
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
)
print(f"Dimension: {inspect.dimension}")
print(f"Suggested models: {[m.label for m in inspect.suggested_models]}")

# Step 2: Standard vector search
result = client.pipeline.query_vectordb(
    query="How do I authenticate with the API?",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",  # must match ingestion model
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
    top_k=5,
)
print(f"Found {result.chunks_retrieved} results (cost: ${result.credits_used:.4f})")
for r in result.results:
    print(f"  [{r.score:.2f}] {r.text[:100]}...")
    # For hierarchical chunks — use parent_text as LLM context:
    if r.metadata.get("is_hierarchical") and r.metadata.get("parent_text"):
        print(f"  → LLM context: {r.metadata['parent_text'][:120]}...")

# Hybrid search — vector + BM25 keyword + Reciprocal Rank Fusion (RRF)
# Best for: exact IDs, error codes, names, multi-hop cross-referencing
result = client.pipeline.query_vectordb(
    query="Who manages Project Alpha?",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="lancedb",
    vector_db_config={"db_path": "./lancedb", "table_name": "docs"},
    top_k=5,
    hybrid_search=True,   # BM25 + vector, merged with RRF
)
for r in result.results:
    print(f"  [rrf={r.rrf_score:.4f}] sources={r.hybrid_sources} — {r.text[:80]}...")
Billing: $0.0002 / chunk retrieved · inspect_vectordb() is always free · Hybrid search: same rate, no surcharge
POST/v1/rag-chat
client.pipeline.rag_chat()

RAG Chat: embed a query, retrieve the most relevant chunks from your vector database, and generate a grounded LLM answer. You bring your own LLM key — scrapedatshi only charges for the vector retrieval.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.rag_chat(
    query="How do I authenticate with the API?",
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",  # must match ingestion model
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
    llm_provider="openai",
    llm_api_key="sk-...",
    llm_model="gpt-4o-mini",
    top_k=5,
)
print(result.answer)
print(f"Based on {result.chunks_retrieved} chunks (cost: ${result.credits_used:.4f})")
for source in result.sources:
    print(f"  [{source.score:.2f}] {source.text[:80]}...")
Billing: $0.0002 / chunk retrieved · LLM tokens are your own cost (not billed by scrapedatshi)
POST/v1/inspect-vectordb
client.pipeline.inspect_vectordb()

Read vector database metadata — dimension, vector count, and suggested embedding models based on the detected dimension. Use this before querying to confirm which embedding model was used during ingestion. Always free — no credits charged.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

result = client.pipeline.inspect_vectordb(
    vector_db="pinecone",
    vector_db_config={
        "api_key": "pc-...",
        "index_host": "https://my-index-abc123.svc.pinecone.io",
    },
)
print(f"Dimension: {result.dimension}")
print(f"Total vectors: {result.total_vector_count:,}")
print(f"Suggested models:")
for model in result.suggested_models:
    print(f"  • {model.label} ({model.provider} / {model.model})")
if result.note:
    print(f"Note: {result.note}")
Billing: Free — no credits deducted
cookies= · headers= · allow_subdomains=

For pages behind a login wall, pass your session cookies and/or custom headers to any fetch method. Credentials are used exclusively on your machine and never transmitted to our servers. The credential shield ensures cookies are only sent to URLs within the permitted domain scope — never leaked to external domains discovered during crawling.

from scrapedatshi import ScrapedatshiClient

client = ScrapedatshiClient()

# Scrape a login-walled page
result = client.pipeline.chunk_url(
    "https://internal.company.com/wiki/api-docs",
    cookies={"session": "abc123", "csrf": "xyz"},
    headers={"Authorization": "Bearer eyJ..."},
)

# Authenticated sitemap crawl
result = client.pipeline.crawl(
    "https://internal.company.com",
    cookies={"session": "abc123"},
    headers={"Authorization": "Bearer eyJ..."},
    max_pages=20,
)

# Spider crawl with subdomain scope
# Crawls company.com + wiki.company.com + docs.company.com
# Cookies only sent to URLs within the same apex domain
result = client.pipeline.crawl(
    "https://company.com",
    crawl_mode="spider",
    cookies={"session": "abc123"},
    allow_subdomains=True,   # multi-part TLDs (.co.uk) handled safely
    max_pages=30,
)

# Full pipeline with authentication
result = client.pipeline.sync(
    url="https://internal.company.com/wiki/api-docs",
    cookies={"session": "abc123"},
    embedding_provider="openai",
    embedding_api_key="sk-...",
    embedding_model="text-embedding-3-small",
    vector_db="pinecone",
    vector_db_config={"api_key": "pc-...", "index_host": "https://..."},
)
Security: Cookies and headers are processed locally on your machine. They are never transmitted to scrapedatshi servers. You are responsible for ensuring your use of session credentials complies with the terms of service of any third-party website you access.
pip install scrapedatshi-mcp

Use all scrapedatshi tools directly inside Claude Desktop — no code required. Just talk to Claude naturally: "Crawl https://docs.example.com and sync it to my Pinecone index using OpenAI embeddings."

Setup

Open Claude Desktop → Settings → Developer → Edit Config, then add: 

{
  "mcpServers": {
    "scrapedatshi": {
      "command": "uvx",
      "args": [
        "--from", "scrapedatshi-mcp[all]",
        "--refresh",
        "scrapedatshi-mcp"
      ],
      "env": {
        "SCRAPEDATSHI_API_KEY": "sds_your_key_here",
        "OPENAI_API_KEY": "sk-...",
        "PINECONE_API_KEY": "pc-..."
      }
    }
  }
}

[all] installs all provider SDKs. --refresh auto-updates on every Claude Desktop restart. Restart Claude Desktop after saving.

Available MCP Tools

scrape_url

Get clean Markdown from a URL

chunk_url

Scrape & chunk a URL into RAG segments

chunk_file

Chunk a local file

crawl_site

Crawl an entire site

extract_data

Extract structured fields

extract_crawl

Multi-page extraction

sync_to_vectordb

URL → embed → inject

ingest_file

File → embed → inject

autorag

Crawl site → embed → inject

inspect_vectordb

Read VDB metadata (free)

query_vectordb

Semantic search

rag_chat

Retrieve + generate answer

verify_provider_key

Verify API key + get models

list_embedding_providers

List embedding providers

list_vector_db_providers

List vector DB providers

get_usage_guide

Tool selection guide

Example conversations

💬 "Scrape https://docs.example.com and show me the content"
💬 "Chunk https://docs.example.com into RAG-ready segments"
💬 "Crawl https://example.com/products and extract the title and price from every page"
💬 "Sync https://docs.example.com to my Pinecone index using OpenAI text-embedding-3-small"
💬 "Crawl the entire docs.stripe.com site and inject it into my Pinecone index"
💬 "Query my Pinecone index for information about API authentication"
💬 "Scrape https://internal.company.com/wiki — use my session cookie: abc123"
PyPI → GitHub →

Supported Providers

Embedding

OpenAI · Cohere · Google Gemini · Mistral · Voyage AI · Ollama (local)

Vector Databases

Pinecone · Qdrant · Supabase (pgvector) · Weaviate · MongoDB Atlas · Azure Cosmos DB · ChromaDB · LanceDB

LLM (extraction + CR)

OpenAI · Anthropic · Google Gemini

Use from scrapedatshi.providers import EMBEDDING_PROVIDERS, VECTOR_DB_PROVIDERS, LLM_PROVIDERS to discover all supported providers and required config fields programmatically.