Extraction & snapshots

Two leaf modules shared by callers on opposite ends of a document's life (ADR-0016): the same extractor that built a document's content at ingest time is what retrieval.citations.resolve_citation calls again to re-derive that text when verifying a citation later. One implementation per format, used from both sides, is what makes "the same extractor that produced it" (ADR-0016 decision 2) hold by construction rather than by two independent implementations kept in sync by hand. Neither module imports the other, and neither is imported back — both follow identity.py's placement precedent: a module outside whichever caller happened to need it first, so sharing it creates no dependency edge between ingestion/ and retrieval/.

Extraction

PdfExtractor and HtmlExtractor are imported by exactly two kinds of caller: the PDF/HTML loaders, to build Document.content at ingest time, and the citation resolver above. Neither pypdf nor beautifulsoup4 is imported at module level — importing groundkit.extraction never requires either optional extra; only constructing the corresponding extractor does.

groundkit.extraction

Deterministic content extraction shared between ingest and citation resolution (ADR-0016 decisions 2, 3; docs/specs/loaders-extracted-and-remote-sources.md §9).

:class:PdfExtractor and :class:HtmlExtractor are imported by exactly two kinds of caller: the Wave 3 PDF/HTML loaders (to build Document.content at ingest time) and :func:groundkit.retrieval.citations.resolve_citation (to re-derive that same text at citation-resolution time). One class per format, used from both sides, is what makes "the same extractor that produced it" (ADR-0016 decision 2) hold by construction rather than by two independent implementations staying in sync by hand.

Placement follows :mod:groundkit.identity's precedent, not */protocols.py's: a module outside whichever caller happened to need it first, so sharing it creates no dependency between ingest and retrieval, and nothing that imports it is imported back. This is a deliberate departure from this repo's other four protocol modules, each of which holds a protocol whose implementations live in the same package as the code that needs them — ExtractorProtocol has no such single owning package.

Neither pypdf nor beautifulsoup4 is imported at module level. Importing this module must never require either optional extra, matching the established pattern in :mod:groundkit.index.dense (_import_lancedb) and :mod:groundkit.retrieval.rerank (_import_cross_encoder).

ExtractorProtocol

Bases: Protocol

Deterministically converts a file's bytes into extracted text.

Satisfied by every extractor this build can run -- both to produce a freshly-ingested Document.content and to re-derive that same text at citation-resolution time (ADR-0016 decision 2). Same bytes in, same text out: no extractor implementation may consult wall-clock time, randomness, environment-dependent library auto-selection, or anything outside the file's own bytes and this object's own pinned configuration.

identity property

identity: str

This extractor's identity string: "<distribution>/<version>".

Derived at runtime from the installed package's own distribution metadata via importlib.metadata.version -- never hardcoded -- so a library upgrade changes this string automatically and a mismatch against a citation's recorded identity (ADR-0016 decision 2) is detected without a separate manual version-tracking step.

extract async

extract(path: Path) -> str

Return the deterministic extracted text for the file at path.

path is already containment-checked by the caller (the loader's own ensure_within_base, or resolve_citation's) -- this method performs no path-safety check of its own.

Raises:

Type	Description
IngestionError	The file cannot be read, or its bytes cannot be parsed as this extractor's format.

Source code in src/groundkit/extraction.py

async def extract(self, path: Path) -> str:
    """Return the deterministic extracted text for the file at ``path``.

    ``path`` is already containment-checked by the caller (the loader's
    own ``ensure_within_base``, or ``resolve_citation``'s) -- this method
    performs no path-safety check of its own.

    Raises:
        IngestionError: The file cannot be read, or its bytes cannot be
            parsed as this extractor's format.
    """
    ...

HtmlExtractor

HtmlExtractor()

Deterministic HTML tag-stripping via BeautifulSoup + html.parser (ADR-0016 decisions 2, 3). Construct via :func:html_extractor, not directly -- see :class:PdfExtractor's docstring for why.

Source code in src/groundkit/extraction.py

def __init__(self) -> None:
    self._identity = f"beautifulsoup4/{importlib.metadata.version('beautifulsoup4')}"

PdfExtractor

PdfExtractor()

Deterministic PDF text extraction via pypdf (ADR-0016 decisions 2, 3).

Construct via :func:pdf_extractor, not directly -- that accessor is what guarantees pypdf already imported successfully before __init__ reads its version, and is what makes every caller share one instance (and therefore one identity computed once, not recomputed per call).

Source code in src/groundkit/extraction.py

def __init__(self) -> None:
    self._identity = f"pypdf/{importlib.metadata.version('pypdf')}"

active_extractors cached

active_extractors() -> Mapping[str, ExtractorProtocol]

Every extractor this process can actually run, keyed by identity string.

Replaces retrieval.citations._ACTIVE_EXTRACTOR_IDENTITIES (a plain frozenset constant, Waves 1-2) with a lazily-memoized accessor. Called on the first extracted-class citation resolved in this process -- never at module import time -- so import groundkit.extraction (and therefore import groundkit.retrieval.citations, which imports it) never requires either extra.

Each candidate is probed independently via :func:pdf_extractor / :func:html_extractor, catching only :class:ConfigurationError (never a broader except): one missing extra must never blank out the other, and a genuine bug inside an installed extractor's construction must not be silently swallowed and reported as "not registered."

Source code in src/groundkit/extraction.py

@lru_cache(maxsize=1)
def active_extractors() -> Mapping[str, ExtractorProtocol]:
    """Every extractor this process can actually run, keyed by identity string.

    Replaces ``retrieval.citations._ACTIVE_EXTRACTOR_IDENTITIES`` (a plain
    frozenset constant, Waves 1-2) with a lazily-memoized accessor. Called on
    the first ``extracted``-class citation resolved in this process -- never
    at module import time -- so ``import groundkit.extraction`` (and
    therefore ``import groundkit.retrieval.citations``, which imports it)
    never requires either extra.

    Each candidate is probed independently via :func:`pdf_extractor` /
    :func:`html_extractor`, catching only :class:`ConfigurationError` (never
    a broader except): one missing extra must never blank out the other, and
    a genuine bug inside an installed extractor's construction must not be
    silently swallowed and reported as "not registered."
    """
    registry: dict[str, ExtractorProtocol] = {}
    for accessor in _EXTRACTOR_ACCESSORS:
        try:
            extractor = accessor()
        except ConfigurationError:
            continue
        registry[extractor.identity] = extractor
    return registry

html_extractor cached

html_extractor() -> HtmlExtractor

The process's one :class:HtmlExtractor, built on first use and memoized.

Raises:

Type	Description
ConfigurationError	The 'html' extra is not installed.

Source code in src/groundkit/extraction.py

@lru_cache(maxsize=1)
def html_extractor() -> HtmlExtractor:
    """The process's one :class:`HtmlExtractor`, built on first use and memoized.

    Raises:
        ConfigurationError: The 'html' extra is not installed.
    """
    _import_bs4()
    return HtmlExtractor()

pdf_extractor cached

pdf_extractor() -> PdfExtractor

The process's one :class:PdfExtractor, built on first use and memoized.

Raises:

Type	Description
ConfigurationError	The 'pdf' extra is not installed.

Source code in src/groundkit/extraction.py

@lru_cache(maxsize=1)
def pdf_extractor() -> PdfExtractor:
    """The process's one :class:`PdfExtractor`, built on first use and memoized.

    Raises:
        ConfigurationError: The 'pdf' extra is not installed.
    """
    _import_pypdf()
    return PdfExtractor()

Snapshots

Pure path arithmetic, no I/O: where a URL-ingested document's local snapshot lives on disk (ADR-0016 decision 4; see also ADR-0023 for when a snapshot is removed, not just where it lives). Written by the URL loader and read by the citation resolver above, so the naming convention that ties a document ID to its snapshot path is asserted from exactly one place, and the two sides cannot independently drift on what it means.

groundkit.snapshots

Where a URL-ingested document's local snapshot lives on disk (ADR-0016 decision 4; docs/specs/loaders-extracted-and-remote-sources.md §10.1). Pure path arithmetic, no I/O -- shared by the URL loader (:class:~groundkit.ingestion.url_loader.UrlLoader, which writes) and :func:groundkit.retrieval.citations.resolve_citation (which reads), so the naming convention is asserted from exactly one place and the two sides cannot independently drift on what a snapshot path means.

Placement follows :mod:groundkit.identity's precedent (see :mod:groundkit.extraction's docstring, which quotes the same reasoning): a module outside whichever caller happened to need it first, so sharing it creates no dependency between ingest and retrieval, and nothing imports it back.

snapshot_dir_for

snapshot_dir_for(index_dir: Path, collection: str) -> Path

The containment root for one collection's stored snapshots.

Sibling of <index_dir>/<collection>.sqlite3 and .lance, following the same per-collection-suffix convention. Not resolved or created here -- the caller (:class:~groundkit.ingestion.url_loader.UrlLoader on the write side, :func:~groundkit.retrieval.citations.resolve_citation on the read side) is responsible for that, exactly as both already are for allowed_base_dir.

Source code in src/groundkit/snapshots.py

def snapshot_dir_for(index_dir: Path, collection: str) -> Path:
    """The containment root for one collection's stored snapshots.

    Sibling of ``<index_dir>/<collection>.sqlite3`` and ``.lance``, following
    the same per-collection-suffix convention. Not resolved or created here
    -- the caller (:class:`~groundkit.ingestion.url_loader.UrlLoader` on the
    write side, :func:`~groundkit.retrieval.citations.resolve_citation` on
    the read side) is responsible for that, exactly as both already are for
    ``allowed_base_dir``.
    """
    return index_dir / f"{collection}.snapshots"

snapshot_path_for

snapshot_path_for(
    snapshot_dir: Path, document_id: str
) -> Path

The snapshot file for one document within its collection's snapshot_dir.

document_id is attacker-influenced in principle (it is a plain string field with no character-class restriction), so it is required here to be a single path component -- one name directly inside snapshot_dir, never a nested path.

That restriction is what makes O_NOFOLLOW at the callers' opens sufficient. O_NOFOLLOW refuses a symlink at the final component only; every intermediate directory is still resolved normally. A document_id of "sub/evil" would pass an :func:~groundkit.utils.path_safety.ensure_within_base check (it escapes nothing) while introducing exactly such an intermediate component, which the write side would then create with mkdir(parents=True) -- reopening the containment-check-to-open race that O_NOFOLLOW closes, one level up where the flag cannot see it. Refusing separators outright removes the intermediate component instead of trying to guard it, so no caller has to reason about the ordering of mkdir against the check.

Callers must still run the result through ensure_within_base (or is_within_base): this function refuses shapes, and containment against a specific root remains the caller's check to make. In production document_id is always Document's own uuid.uuid4().hex default, which is 32 hex characters and can never contain a separator -- this guard exists so that property is enforced rather than assumed.

What counts as a separator is what the running platform treats as one (os.sep/os.altsep, plus a drive prefix). A backslash is an ordinary filename character on POSIX and a separator on Windows, so the same document_id is legitimately a single component on one and a nested path on the other; branching on the platform's own notion states that rather than hard-coding one platform's answer.

Raises:

Type	Description
ValueError	document_id is empty, is . or .., contains a null byte, contains a path separator, or carries a drive prefix. ValueError specifically because that is what ensure_within_base raises for a rejected path, so the callers' existing handlers cover this without a second except clause.

Source code in src/groundkit/snapshots.py

def snapshot_path_for(snapshot_dir: Path, document_id: str) -> Path:
    """The snapshot file for one document within its collection's ``snapshot_dir``.

    ``document_id`` is attacker-influenced in principle (it is a plain string
    field with no character-class restriction), so it is required here to be a
    single path component -- one name directly inside ``snapshot_dir``, never a
    nested path.

    That restriction is what makes ``O_NOFOLLOW`` at the callers' opens
    sufficient. ``O_NOFOLLOW`` refuses a symlink at the *final* component only;
    every intermediate directory is still resolved normally. A ``document_id``
    of ``"sub/evil"`` would pass an
    :func:`~groundkit.utils.path_safety.ensure_within_base` check (it escapes
    nothing) while introducing exactly such an intermediate component, which
    the write side would then create with ``mkdir(parents=True)`` -- reopening
    the containment-check-to-open race that ``O_NOFOLLOW`` closes, one level
    up where the flag cannot see it. Refusing separators outright removes the
    intermediate component instead of trying to guard it, so no caller has to
    reason about the ordering of ``mkdir`` against the check.

    Callers must still run the result through ``ensure_within_base`` (or
    ``is_within_base``): this function refuses *shapes*, and containment
    against a specific root remains the caller's check to make. In production
    ``document_id`` is always ``Document``'s own ``uuid.uuid4().hex`` default,
    which is 32 hex characters and can never contain a separator -- this guard
    exists so that property is enforced rather than assumed.

    What counts as a separator is what the **running platform** treats as one
    (``os.sep``/``os.altsep``, plus a drive prefix). A backslash is an
    ordinary filename character on POSIX and a separator on Windows, so the
    same ``document_id`` is legitimately a single component on one and a
    nested path on the other; branching on the platform's own notion states
    that rather than hard-coding one platform's answer.

    Raises:
        ValueError: ``document_id`` is empty, is ``.`` or ``..``, contains a
            null byte, contains a path separator, or carries a drive prefix.
            ``ValueError`` specifically because that is what
            ``ensure_within_base`` raises for a rejected path, so the callers'
            existing handlers cover this without a second except clause.
    """
    if not document_id:
        raise ValueError("a snapshot document_id must not be empty")
    if document_id in {".", ".."}:
        raise ValueError(
            f"snapshot document_id {document_id!r} is a directory reference, not a file name"
        )
    if "\x00" in document_id:
        raise ValueError("a snapshot document_id must not contain a null byte")
    if any(separator in document_id for separator in _SEPARATORS):
        raise ValueError(
            f"snapshot document_id {document_id!r} contains a path separator; it must name "
            "a single file directly inside the snapshot directory, so that no intermediate "
            "path component exists for a symlink to be planted at"
        )
    if os.path.splitdrive(document_id)[0]:
        raise ValueError(
            f"snapshot document_id {document_id!r} carries a drive prefix; it must name a "
            "single file directly inside the snapshot directory"
        )
    return snapshot_dir / document_id