Explainer
Web Scraping for RAG Needs More Than Clean Markdown
Before you index scraped pages for RAG, settle permission, source metadata, validation, retrieval checks and how pages get refreshed or deleted.
I would build a RAG scraping pipeline around traceable documents, not just a folder of clean Markdown. Start with allowed targets, preserve source and extraction metadata, validate the content, then chunk and index it so an answer can point back to the exact material it used.
My first buying decision is whether the source needs a browser, a bounded crawl or known-field extraction. My first acceptance decision is whether the right document survives ingestion with enough evidence to support a citation. Formatting and retrieval are separate checks.
Short on time?
Here's my take
- Where to start
- Allowed scope and provenance
- Decide which documents may be fetched and keep their source/version context before generating chunks.
- Before you index anything
- Validate before indexing
- Check the page, retained content, units and duplicates; a parsed or attractive output is not the whole test.
- Once it is running
- Refresh and deletion records
- Preserve change history so outdated or removed documents do not quietly remain available to retrieval.
Prices verified . Documentation checked 11 November 2026. This is a proposed workflow, with no crawler or retrieval performance results.
What does the whole pipeline look like? #
This table is my proposed workflow; it describes design choices rather than a completed system. LlamaIndex’s documents/nodes guidance and LangChain’s splitter documentation inform the structure; the checks are the acceptance criteria I would attach to an implementation. Documents and nodes, splitters.
| Step | Output I would keep | Decision before moving on |
|---|---|---|
| Permission and scope | Allowed source list and access rules | Is this target permitted and bounded? |
| Fetch/render | Source document, URL and fetch metadata | Is this the intended page rather than an error or login screen? |
| Clean/deduplicate | Retained content and a source hash | Did important sections survive, and is this document already present? |
| Chunk | Section-aware chunks with source metadata | Can each chunk be traced to the correct source/version? |
| Embed/index | Index entries keyed to those chunks | Can updates/deletions replace the correct records? |
| Retrieve/cite | Selected evidence and answer citations | Does the cited material support the actual answer? |
| Refresh/delete | Change records and tombstones | Can stale or removed material be withdrawn consistently? |
What are you allowed to crawl? #
I would establish the target set and permission before crawling. Robots is a separate protocol rather than access authorization; an absent disallow rule does not settle your access rights. RFC 9309.
Then configure bounds such as the allowed host/path set, page count and traversal depth for your intended corpus. Crawl4AI documents bounded deep-crawling strategies, and Firecrawl documents crawl scope. Those are useful controls to inspect, not proof that every possible configuration has the same limits. Crawl4AI traversal, Firecrawl crawl. If those two are your shortlist, Crawl4AI vs Firecrawl compares them as self-hosted and managed options.
Defaults differ. Crawl4AI’s check_robots_txt defaults false, Firecrawl crawl respects robots by default, and Scrapy’s docs distinguish generated-project settings from the historical fallback. I would explicitly set the policy rather than infer it from the library name. Crawl4AI parameters, Firecrawl, Scrapy settings.
Does the source need a browser? #
I would use the simplest documented route that preserves the needed content. A framework such as Crawlee offers HTTP/browser crawling, while extraction utilities such as Trafilatura address text and metadata within a pipeline. Neither description establishes which route your documents need without checking them. Crawlee repository, Trafilatura usage. Both sit in my roundup of open-source web crawlers for LLMs, and search, fetch or browser helps you decide how much machinery a source really needs.
Define the output before buying a page allowance. If you need fields as well as text, Firecrawl’s JSON mode has additional credit costs; optional modifiers change the budget too. JSON docs, billing.
For format choices, read website-to-Markdown options and Firecrawl versus Jina Reader. Include the operating work of the chosen fetch/render route in that decision.
Can you trace every chunk back to its page? #
I would keep enough source metadata to reconstruct where a chunk came from. LlamaIndex distinguishes documents and nodes and supports metadata around that structure; the following fields are my proposed contract, not a universal vendor schema. Documents and nodes.
My proposed record includes source URL, fetch time, raw-content hash, extraction version, section heading and access scope. Each derived chunk should point back to that source/version record, so changing a splitter does not make the original evidence disappear.
I would retain the raw source where permitted by the applicable access and retention rules, alongside the cleaned representation. When a table value or code block goes missing, that relationship helps identify the stage that changed it.
The source hash is an identifier for comparison in this design, not a proof of the page’s truth. I would still review whether the extracted content matches the intended document.
Check the content before you chunk it #
I would check retained content before turning it into many index entries. For known fields, Crawl4AI’s deterministic strategies can supply a clear selector-based route; a shape validator can then catch malformed fields. Neither step establishes factual correctness by itself. Deterministic extraction, JSON Schema.
My designed checks would include headings, code whitespace, tables, currency/units, missing fields and the expected page type. Those are proposed cases, not defects measured in a particular crawler.
I would also deduplicate before indexing repeated source versions. A proposed policy might keep one canonical source/version record and attach each chunk to it, while retaining the change log needed for later refreshes.
For structured records, use the extraction decision guide. It explains why a valid JSON object can still contain a wrong currency or evidence from the wrong page.
How should you split pages into chunks? #
I would choose a splitter that fits the document and model constraints, then evaluate the intended questions. LangChain documents token-, character- and document-oriented splitting choices; LlamaIndex’s node structure helps attach metadata to derived units. Splitter documentation, documents/nodes.
This research supplies no universal best chunk size. I would not publish a fixed number as a performance recommendation without an evaluation on the documents and questions involved.
My evaluation would inspect the retrieved evidence and its source context before accepting the final answer. A nicely formatted page or successful ingestion call is useful, but it does not by itself establish that the answer’s citation supports the claim.
For example, an illustrative product question needs the relevant product section and its unit/billing context. I would reject a citation that points only to a navigation menu or a different product, even if the answer is phrased confidently.
Scraped text can carry prompt injections #
I would prevent source text from redefining the task or its permissions. OWASP identifies indirect prompt injection through untrusted content; retrieved web material can carry such instructions alongside otherwise useful facts. Prompt-injection guidance.
In my proposed design, retrieved content supplies evidence, while the system controls tools, destination permissions and the extraction task. I would log the original source and quarantine unexpected instructions rather than silently letting a page expand the crawl or invoke another service.
That is also why I would separate discovery from retrieval. Firecrawl versus Tavily helps distinguish search and page-fetching decisions; discovery does not replace the source and permission checks at ingestion.
Plan for pages that change or vanish #
I would attach a refresh policy and deletion path to every source/version record. The desired behavior is explicit: detect a changed source, update the affected chunks/index entries, and record what happened before accepting fresh citations.
For a removed document, my proposed policy uses a tombstone and removes the affected retrieval records. I would test that the answer path no longer selects those chunks instead of treating a successful delete response from one component as the whole result.
For a changed document, preserve the prior version’s change record and revalidate the relevant content before re-embedding. The exact schedule depends on your source and obligations; this research has not established a universally correct refresh interval.
Which crawler should you choose for RAG? #
My recommendation is to choose the fetch/extraction route after defining permission, output and provenance requirements. Use a browser-capable route only where the source calls for it; prefer deterministic fields when their structure is known, and price model-generated output separately.
I would accept the pipeline only when an allowed-source fixture survives the stages, its citations are traceable, and refresh/deletion works in an isolated check. None of those checks was run for this article.
Start with the crawl API hub, then structured extraction for field contracts. If tools are exposed to an assistant, the web scraping MCP guide adds a separate connection, permission and billing decision.
FAQ #
Does robots.txt mean I’m allowed to scrape? #
No. The robots protocol is not access authorization. Establish the actual allowed scope separately. RFC 9309.
Is Markdown enough for a good RAG pipeline? #
I would also require source/version metadata, content checks, retrieval evaluation and supported citations. This article has not measured a formatting-to-answer-quality improvement.
What’s the best chunk size for RAG? #
There is no measured universal choice here. Use the splitter options to design a configuration, then evaluate your intended questions.
What happens when a page changes or disappears? #
My proposed policy updates or withdraws the affected records and verifies the retrieval path afterward. It is an acceptance requirement, not a completed implementation or every vendor’s default behavior.
Sources
14 sources, accessed .
Official pricing, docs, changelogs and status pages 13
- Modifiers, failures, rollover and limits
- Scope, robots and result expiry
- v2 single-page JSON extraction
- Robots and browser/run configuration
- BFS/DFS/best-first bounded traversal
- CSS/XPath and deterministic extraction
- Robots/project defaults and concurrency
- Fetch/extract API and output choices
- Output-shape validation vocabulary
- Token/character/document splitting
- Documents, metadata and nodes
- Untrusted retrieved-content instructions
- Robots protocol is not access authorization
GitHub code, issues and releases 1
Last updated: 13 October 2026. Researched and drafted with AI tools; human editorial review and publication approval are pending.
