Context retrieval for RAG over web pages, from fetch to footnote

1. 1

   Step 1: Fetch clean Markdown with metadata

   Start from a reviewed list of public URLs. Keep the main content as Markdown plus the final URL, title, published date when present and retrieval time.

2. 2

   Step 2: Chunk by headings, keep the path

   Split on headings first, cap long sections, keep tables and code blocks whole, and store the page title and heading path on every chunk.

3. 3

   Step 3: Retrieve, rerank, measure

   Combine keyword and vector search, rerank the top candidates and check recall on a small set of real questions before you tune anything else.

4. 4

   Step 4: Cite the passage, refresh the source

   Cite the final URL and the section the answer came from. Re-fetch each source on its own schedule, re-embed what changed and delete what is gone.

import os, requests
from langchain_text_splitters import MarkdownHeaderTextSplitter

urls = ["https://example.com/docs/a", "https://example.com/docs/b"]
r = requests.post("https://livewebdata.com/api/v1/contents", timeout=90,
    headers={"Authorization": f"Bearer {os.environ['LIVE_WEB_DATA_API_KEY']}"},
    json={"urls": urls,
          "text": {"maxCharacters": 100000}}).json()

splitter = MarkdownHeaderTextSplitter([("#", "h1"), ("##", "h2"), ("###", "h3")])
pages = {p["id"]: p for p in r["results"]}  # id = the requested URL
chunks, failed = [], []
for st in r["statuses"]:  # one status per requested URL
    if st["status"] != "success":
        failed.append((st["id"], st["error"]["tag"]))
        continue
    page = pages[st["id"]]
    for i, doc in enumerate(splitter.split_text(page["markdown"])):
        chunks.append({
            "id": f"{page['url']}#{i}", "text": doc.page_content,
            "heading_path": list(doc.metadata.values()),
            "url": page["url"], "title": page["title"],
            "retrieved_at": st["fetchedAt"],
        })

Markdown is cut at text.maxCharacters, 100,000 characters by default. The splitter is LangChain's; any heading-aware one works. Chunking and storage stay yours.

• Web content ingestion for RAG: what to keep after the fetch

  The source list, metadata schema, document IDs and failure records that decide what your retriever can find and what your model can cite.

• Chunking Markdown for RAG: structure first, size second

  Split web Markdown on its headings, keep tables and code whole, and give every chunk the page title and heading path it needs to be found.

• RAG retrieval for web pages: evaluate, hybrid search, rerank

  A 30-question golden set, then contextual chunk headers, hybrid search, reranking and context packing, each tested on your own pages.

• RAG citations from web sources: cite the passage, URL and date

  Cite the final URL, the passage and the date it was read, verify every citation after generation, and handle pages that change or disappear.

• RAG index freshness: re-fetch, re-embed, delete what is gone

  Refresh each source on its own schedule, re-embed only what changed, delete what is gone, and know how old every answer's sources are.

• November 4, 2026 · Freshness

  Delete stale vectors without deleting pages that only failed once

  Removed pages keep answering until their vectors go. Delete on a confirmed 404 or 410, never on a timeout, and watch sync modes that delete what is missing.

• October 31, 2026 · Retrieval

  BM25 vs embeddings on documentation pages: which queries each wins

  BM25 finds exact strings such as error codes and versions; embeddings find paraphrases. On docs pages you usually need both, fused into one ranking.

• October 27, 2026 · Chunking

  Chunk size for RAG: a starting point and one tuning pass

  There is no best chunk size for RAG. Start from headings with a cap, count in tokens, and change one setting at a time against real questions.