Retrieval

Search orchestration, reciprocal-rank fusion, cross-encoder rerank, and citation resolution. This is the deterministic core: no LLM runs on any path in this package.

Search

groundkit.retrieval.search

BM25 + dense + hybrid search returning citation-bearing results.

Deterministic — no LLM in this path (SPEC.md §2). Phase 1 shipped the BM25 path; Phase 3 Wave C adds the dense read path and RRF hybrid fusion behind the same surface, with the document-source join happening exactly once, on the surviving results (ADR-0006). Rerank arrives in Wave D.

Retriever

Retriever(
    store: MetadataStoreProtocol,
    bm25: LexicalIndexProtocol,
    config: RetrievalConfig | None = None,
    *,
    embedder: EmbeddingProtocol | None = None,
    vector_store: VectorStoreProtocol | None = None,
    documents_at_open: frozenset[str] | None = None,
    collection_is_dense_bound: bool = False,
    collection: str | None = None,
)

Search a persisted collection, resolving every hit to a citation.

Construct via :meth:open, which rebuilds the in-memory BM25 index from the metadata store (ADR-0002) and — when a dense pair is supplied — verifies the collection's embedding-identity manifest (ADR-0004 decision 3) and captures the open()-time document snapshot the dense staleness filter is defined over.

A Retriever reflects the store's state at open() time only and is never refreshed — it must be reopened after any write to the store (re-ingest, edit, deletion) to observe that write. On the BM25 side that is structural: the in-memory index is a rebuild, so content ingested after open() has no representation in it at all and silently returns zero results, while a hit against modified or deleted content fails closed (:class:~groundkit.errors.RetrievalError, per the index inconsistency check in :meth:search). The dense side reads a live vector-store handle, so the same snapshot semantics are enforced by filtering instead of by rebuild: dense hits are restricted to documents that existed at open() time, making content ingested afterwards silently invisible on both paths in exactly the same way. Both paths fail closed identically on a hit whose document has no stored source — on the dense side that includes orphaned vectors, i.e. rows whose document a BM25-only Indexer deleted from SQLite without a vector store to delete from, or the residue of an interrupted ingest (KNOWN_LIMITATIONS.md). Stale content cannot be resolved through the dense path at all: a dense-enabled indexer deletes a replaced document's vectors in lockstep (ADR-0004 decision 6), and the fail-closed join covers the case where it did not. See ADR-0002's Consequences section for the full staleness discussion.

Source code in src/groundkit/retrieval/search.py

def __init__(
    self,
    store: MetadataStoreProtocol,
    bm25: LexicalIndexProtocol,
    config: RetrievalConfig | None = None,
    *,
    embedder: EmbeddingProtocol | None = None,
    vector_store: VectorStoreProtocol | None = None,
    documents_at_open: frozenset[str] | None = None,
    collection_is_dense_bound: bool = False,
    collection: str | None = None,
) -> None:
    _validate_dense_pair(embedder, vector_store)
    if embedder is not None and documents_at_open is None:
        raise ConfigurationError(
            "A dense-paired Retriever requires documents_at_open — the open()-time "
            "document snapshot its staleness filter is defined over. Construct via "
            "Retriever.open(), which captures it."
        )
    self._store = store
    self._bm25 = bm25
    self._config = config or RetrievalConfig()
    self._embedder = embedder
    self._vector_store = vector_store
    self._documents_at_open: frozenset[str] = (
        documents_at_open if documents_at_open is not None else frozenset()
    )
    self._collection_is_dense_bound = collection_is_dense_bound
    # Telemetry label only (ADR-0022 decision 5), never read back by this
    # class. There is no MetadataStoreProtocol accessor for a collection's
    # own name (see the class docstring's snapshot discussion for the
    # analogous reasoning on documents_at_open) — widening that protocol
    # for an observability nicety is out of proportion, so a caller that
    # knows the name (CollectionRuntime, which opened the store from it)
    # passes it through here instead.
    self._collection = collection

open async classmethod

open(
    store: MetadataStoreProtocol,
    config: RetrievalConfig | None = None,
    *,
    embedder: EmbeddingProtocol | None = None,
    vector_store: VectorStoreProtocol | None = None,
    collection: str | None = None,
) -> Retriever

Build a retriever over store, rebuilding BM25 from persisted chunks.

With a dense pair supplied, the collection's embedding-identity manifest is verified first — before the O(corpus) BM25 rebuild does any work — closing the second of ADR-0004 decision 3's two boundaries (Indexer closed the ingest boundary in Wave B). A mismatch is :class:~groundkit.errors.IndexIdentityError, never a re-embed and never a fallback. A collection with no manifest (never dense-ingested) verifies trivially here — opening is legitimate, and a caller may then search "bm25" — but a dense or hybrid search against it is refused in :meth:search (ADR-0008), because there are no vectors for either mode to read.

That verification is the single source of the dense-bound verdict, which is why verify_manifest returns the manifest it checked instead of this method reading it again afterwards. Deciding "is this collection dense-bound" from a later read than the one the identity was checked against is a TOCTOU hole with a silent wrong answer at the end of it: an unbound collection passes the identity check trivially, and if a concurrent dense ingest (another task, or another process — the store's lock spans neither) binds it to a different provider before the later read, that read reports "bound" and this retriever proceeds to answer dense and hybrid queries by matching this embedder's query vectors against that provider's index. Mixed embedding spaces, the exact corruption ADR-0004 exists to make unrepresentable, with no error anywhere. One read decides both, so the two answers cannot disagree.

The residual window is deliberately biased closed: a collection bound after this read is treated as unbound, so dense and hybrid are refused rather than answered. That is also the already-documented snapshot semantics — a retriever never observes writes that land after open(), and a dense side that did not exist at open() is no exception.

Parameters:

Name	Type	Description	Default
store	MetadataStoreProtocol	The collection's metadata store.	required
config	RetrievalConfig | None	Retrieval settings (defaults if omitted).	None
embedder	EmbeddingProtocol | None	Optional embedding provider for the dense read path (keyword-only; both or neither with vector_store). Used to embed queries; its (provider, model_name, dimensions) triple is verified against the collection manifest.	None
vector_store	VectorStoreProtocol | None	Optional dense vector store (keyword-only).	None
collection	str | None	Optional collection name (keyword-only), attached to every span this retriever's :meth:search opens as the groundkit.collection attribute (ADR-0022 decision 5). Stored as-is on self._collection; never validated against store here, since the store carries no collection-name accessor to validate it against (see :meth:__init__).	None

Returns:

Type	Description
Retriever	A ready-to-search :class:Retriever.

Raises:

Type	Description
ConfigurationError	Exactly one of embedder / vector_store was supplied.
IndexIdentityError	The collection is bound to a different embedding identity (ADR-0004).
StorageError	The collection is manifest-bound and holds documents, but its vector store is empty — a lost dense side, refused here rather than silently answering every dense and hybrid query from an empty index. See :func:~groundkit.index.dense.verify_dense_side_present.

Source code in src/groundkit/retrieval/search.py

@classmethod
async def open(
    cls,
    store: MetadataStoreProtocol,
    config: RetrievalConfig | None = None,
    *,
    embedder: EmbeddingProtocol | None = None,
    vector_store: VectorStoreProtocol | None = None,
    collection: str | None = None,
) -> Retriever:
    """Build a retriever over ``store``, rebuilding BM25 from persisted chunks.

    With a dense pair supplied, the collection's embedding-identity
    manifest is verified *first* — before the O(corpus) BM25 rebuild does
    any work — closing the second of ADR-0004 decision 3's two
    boundaries (``Indexer`` closed the ingest boundary in Wave B). A
    mismatch is :class:`~groundkit.errors.IndexIdentityError`, never a
    re-embed and never a fallback. A collection with no manifest (never
    dense-ingested) verifies trivially here — opening is legitimate, and
    a caller may then search ``"bm25"`` — but a ``dense`` or ``hybrid``
    search against it is refused in :meth:`search` (ADR-0008), because
    there are no vectors for either mode to read.

    **That verification is the single source of the dense-bound
    verdict**, which is why ``verify_manifest`` returns the manifest it
    checked instead of this method reading it again afterwards. Deciding
    "is this collection dense-bound" from a *later* read than the one
    the identity was checked against is a TOCTOU hole with a silent
    wrong answer at the end of it: an unbound collection passes the
    identity check trivially, and if a concurrent dense ingest (another
    task, or another process — the store's lock spans neither) binds it
    to a different provider before the later read, that read reports
    "bound" and this retriever proceeds to answer ``dense`` and
    ``hybrid`` queries by matching *this* embedder's query vectors
    against *that* provider's index. Mixed embedding spaces, the exact
    corruption ADR-0004 exists to make unrepresentable, with no error
    anywhere. One read decides both, so the two answers cannot disagree.

    The residual window is deliberately biased closed: a collection
    bound *after* this read is treated as unbound, so ``dense`` and
    ``hybrid`` are refused rather than answered. That is also the
    already-documented snapshot semantics — a retriever never observes
    writes that land after ``open()``, and a dense side that did not
    exist at ``open()`` is no exception.

    Args:
        store: The collection's metadata store.
        config: Retrieval settings (defaults if omitted).
        embedder: Optional embedding provider for the dense read path
            (keyword-only; both or neither with ``vector_store``). Used
            to embed queries; its ``(provider, model_name, dimensions)``
            triple is verified against the collection manifest.
        vector_store: Optional dense vector store (keyword-only).
        collection: Optional collection name (keyword-only), attached
            to every span this retriever's :meth:`search` opens as the
            ``groundkit.collection`` attribute (ADR-0022 decision 5).
            Stored as-is on ``self._collection``; never validated
            against ``store`` here, since the store carries no
            collection-name accessor to validate it against (see
            :meth:`__init__`).

    Returns:
        A ready-to-search :class:`Retriever`.

    Raises:
        ConfigurationError: Exactly one of ``embedder`` /
            ``vector_store`` was supplied.
        IndexIdentityError: The collection is bound to a different
            embedding identity (ADR-0004).
        StorageError: The collection is manifest-bound and holds
            documents, but its vector store is empty — a lost dense
            side, refused here rather than silently answering every
            dense and hybrid query from an empty index. See
            :func:`~groundkit.index.dense.verify_dense_side_present`.
    """
    cfg = config or RetrievalConfig()
    _validate_dense_pair(embedder, vector_store)
    manifest: CollectionManifest | None = None
    if embedder is not None:
        # Verdict and manifest from one read: see the docstring for the
        # mixed-embedding-space race a second read would open.
        manifest = await store.verify_manifest(identity_of(embedder))
    if embedder is not None and vector_store is not None:
        await verify_dense_side_present(
            store, vector_store, embedder.dimensions, manifest=manifest
        )
    bm25 = await BM25Index.from_store(store, k1=cfg.bm25_k1, b=cfg.bm25_b)
    documents_at_open: frozenset[str] | None = None
    if embedder is not None:
        documents_at_open = frozenset(await store.get_document_sources())
    # Recorded, not enforced, here. A collection with no manifest was
    # never dense-ingested, which is a legitimate state to open with a
    # dense pair — the caller may only ever search "bm25". It is
    # searching *dense* or *hybrid* against it that cannot be answered,
    # so the refusal lives in search(), per mode.
    dense_bound = manifest is not None
    logger.info("Retriever opened over %d chunks", bm25.size)
    return cls(
        store,
        bm25,
        cfg,
        embedder=embedder,
        vector_store=vector_store,
        documents_at_open=documents_at_open,
        collection_is_dense_bound=dense_bound,
        collection=collection,
    )

search async

search(
    query: str,
    top_k: int | None = None,
    *,
    mode: SearchMode = "bm25",
) -> SearchResponse

Search the collection.

The default mode is "bm25" deliberately, and stays so under ADR-0007 even though Wave E's measured eval delta favoured hybrid on every retrieval-quality metric. The reason is abstention: "hybrid" applies no score_threshold (ADR-0005 decision 6 — see below), so no configuration makes it return nothing for a question the corpus cannot answer, and a default that always answers is the wrong default for citation-verifiable retrieval. "dense" does honour score_threshold; what it lacks is a measured, defensible value.

score_threshold applies to the producer-scored modes (bm25, dense) and never to hybrid: fused scores are rank-derived quantities an absolute threshold would silently reinterpret, so ADR-0005 decision 6 excludes them — thresholding the pre-fusion candidate lists would reintroduce the same threshold by a side door, so those are not thresholded either.

Parameters:

Name	Type	Description	Default
query	str	Non-empty query text.	required
top_k	int | None	Result cap for this call; defaults to config. Must be in 1..MAX_TOP_K.	None
mode	SearchMode	"bm25" (lexical only, the default), "dense" (vector only), or "hybrid" (RRF fusion of both, ADR-0005; the response's metadata["stage"] reports it as "fusion").	'bm25'

Returns:

Name	Type	Description
A	SearchResponse	class:SearchResponse whose results each carry a resolvable
	SearchResponse	citation (source + character offsets), with
	SearchResponse	metadata["stage"] naming the stage that produced them
honestly	SearchResponse	"bm25", "dense", or "fusion".

Raises:

Type	Description
RetrievalError	Empty/whitespace query, out-of-range top_k, or an index inconsistency (a hit whose document has no stored source — including orphaned dense vectors — fail closed rather than emit an unverifiable citation).
ConfigurationError	mode is "dense" or "hybrid" but this retriever was opened without a dense pair, or the collection has no embedding-identity manifest and therefore no vectors either mode could read (ADR-0008). The second case is refused rather than answered from an empty dense side, which would return lexical results labelled as dense or fused.

Wrapped in one OTel span, groundkit.retrieve.search (SPEC.md §3, ADR-0022 decision 5), covering the whole call — validation, the store read, and whichever mode branch runs. Only allowlisted attributes ever reach it: collection name, retrieval mode, stage, top_k, result count, latency, and — on any failure — the raised exception's type name as a typed failure code. query itself is never attached, at any point in this method or anything it calls — SPEC.md §3 and ADR-0022 decision 3 forbid query text on a span explicitly, and this is the one property tests/test_span_instrumentation.py exists to pin down with a sentinel value.

Source code in src/groundkit/retrieval/search.py

async def search(
    self,
    query: str,
    top_k: int | None = None,
    *,
    mode: SearchMode = "bm25",
) -> SearchResponse:
    """Search the collection.

    The default mode is ``"bm25"`` deliberately, and stays so under
    ADR-0007 even though Wave E's measured eval delta favoured hybrid on
    every retrieval-quality metric. The reason is abstention: ``"hybrid"``
    applies no ``score_threshold`` (ADR-0005 decision 6 — see below), so
    no configuration makes it return nothing for a question the corpus
    cannot answer, and a default that always answers is the wrong default
    for citation-verifiable retrieval. ``"dense"`` *does* honour
    ``score_threshold``; what it lacks is a measured, defensible value.

    ``score_threshold`` applies to the producer-scored modes (``bm25``,
    ``dense``) and never to hybrid: fused scores are rank-derived
    quantities an absolute threshold would silently reinterpret, so
    ADR-0005 decision 6 excludes them — thresholding the pre-fusion
    candidate lists would reintroduce the same threshold by a side door,
    so those are not thresholded either.

    Args:
        query: Non-empty query text.
        top_k: Result cap for this call; defaults to config. Must be in
            ``1..MAX_TOP_K``.
        mode: ``"bm25"`` (lexical only, the default), ``"dense"``
            (vector only), or ``"hybrid"`` (RRF fusion of both,
            ADR-0005; the response's ``metadata["stage"]`` reports it
            as ``"fusion"``).

    Returns:
        A :class:`SearchResponse` whose results each carry a resolvable
        citation (source + character offsets), with
        ``metadata["stage"]`` naming the stage that produced them
        honestly: ``"bm25"``, ``"dense"``, or ``"fusion"``.

    Raises:
        RetrievalError: Empty/whitespace query, out-of-range ``top_k``,
            or an index inconsistency (a hit whose document has no
            stored source — including orphaned dense vectors — fail
            closed rather than emit an unverifiable citation).
        ConfigurationError: ``mode`` is ``"dense"`` or ``"hybrid"`` but
            this retriever was opened without a dense pair, or the
            collection has no embedding-identity manifest and therefore
            no vectors either mode could read (ADR-0008). The second
            case is refused rather than answered from an empty dense
            side, which would return lexical results labelled as dense
            or fused.

    Wrapped in one OTel span, ``groundkit.retrieve.search`` (SPEC.md §3,
    ADR-0022 decision 5), covering the whole call — validation, the
    store read, and whichever mode branch runs. Only allowlisted
    attributes ever reach it: collection name, retrieval mode, stage,
    ``top_k``, result count, latency, and — on any failure — the raised
    exception's type name as a typed failure code. **``query`` itself
    is never attached, at any point in this method or anything it
    calls** — SPEC.md §3 and ADR-0022 decision 3 forbid query text on a
    span explicitly, and this is the one property
    ``tests/test_span_instrumentation.py`` exists to pin down with a
    sentinel value.
    """
    with tracer.start_as_current_span("groundkit.retrieve.search") as span:
        try:
            if not query.strip():
                raise RetrievalError("Query must not be empty")
            k = top_k if top_k is not None else self._config.top_k
            if not 1 <= k <= MAX_TOP_K:
                raise RetrievalError(f"top_k must be between 1 and {MAX_TOP_K}, got {k}")

            started = time.perf_counter_ns()
            # One lookup object per search, never per retriever: the
            # document read has to stay live so a document deleted after
            # open() fails closed instead of resolving against a snapshot
            # (see :class:`_DocumentRecordLookup`).
            lookup = _DocumentRecordLookup(self._store)

            metadata: dict[str, object]
            stage: Stage
            if mode == "bm25":
                pairs = await self._bm25_search(query, k)
                results = await self._resolve(pairs, lookup, apply_threshold=True)
                stage = "bm25"
                metadata = {"stage": stage, "top_k": k}
            elif mode == "dense":
                embedder, vector_store = self._require_dense(mode)
                pairs = await self._dense_candidates(query, k, lookup, embedder, vector_store)
                results = await self._resolve(pairs, lookup, apply_threshold=True)
                stage = "dense"
                metadata = {"stage": stage, "top_k": k}
            elif mode == "hybrid":
                embedder, vector_store = self._require_dense(mode)
                bm25_pairs = await self._bm25_search(query, k)
                dense_pairs = await self._dense_candidates(
                    query, k, lookup, embedder, vector_store
                )
                fused = reciprocal_rank_fusion(
                    [bm25_pairs, dense_pairs], rrf_k=self._config.rrf_k, top_k=k
                )
                results = await self._resolve(fused, lookup, apply_threshold=False)
                stage = "fusion"
                metadata = {"stage": stage, "top_k": k, "rrf_k": self._config.rrf_k}
            else:
                # SearchMode is a type hint, not a runtime guard. Falling
                # through to hybrid would answer a typo'd mode with fused
                # results and stamp metadata["stage"] = "fusion" on them —
                # a wrong answer reported as a valid one, which SPEC.md
                # §2's fail-closed rule forbids. An unknown mode is a
                # caller bug; name it.
                raise RetrievalError(
                    f"Unknown search mode {mode!r}; expected one of 'bm25', 'dense', 'hybrid'"
                )

            latency_ms = (time.perf_counter_ns() - started) / _NS_PER_MS
            logger.info("Search returned %d results in %.2f ms", len(results), latency_ms)
            response = SearchResponse(
                query=query,
                results=results,
                total_results=len(results),
                metadata={**metadata, "latency_ms": latency_ms},
            )
        except Exception as exc:
            # Telemetry only: this except exists to label the span, never
            # to handle the failure. The bare `raise` below re-raises the
            # exact exception, unwrapped.
            span.set_attributes(
                span_attributes(collection=self._collection, failure_kind=type(exc).__name__)
            )
            raise
        span.set_attributes(
            span_attributes(
                collection=self._collection,
                retrieval_mode=mode,
                stage=stage,
                top_k=k,
                result_count=len(results),
                duration_ms=latency_ms,
            )
        )
        return response

Fusion

groundkit.retrieval.fusion

Reciprocal-rank fusion of lexical and dense result lists (Phase 3, ADR-0005).

Pure code: no I/O, no async, no logging, and no imports beyond stdlib and :class:~groundkit.contracts.Chunk (imported only under TYPE_CHECKING, exactly as index/bm25.py does). This module never imports groundkit.config, groundkit.errors, or any store — invalid arguments raise ValueError directly, because this is a pure-math seam, not a pipeline stage.

Every decision below is ADR-0005's, cited here rather than re-derived:

• Rank, not score (decision 1). BM25 scores are unbounded and corpus-dependent; cosine similarity is bounded but its distribution depends on the embedding model. Neither is comparable to the other, so fusing by score would need an unprincipled normalization. RRF instead consumes only each ranking's permutation: RRFscore(d) = Σ_{r∈R} 1 / (k + r(d)). Input scores are therefore ignored entirely — only a chunk's 1-based position within each ranking matters.
• rrf_k is a caller-supplied parameter; this module owns no default (decision 2). The value 60 is pinned as RetrievalConfig.rrf_k's default (config.py), and its provenance — the original RRF paper's pilot investigation, corroborated by Elasticsearch's rank_constant default — is recorded in ADR-0005, not here. :func:reciprocal_rank_fusion takes rrf_k as a required keyword argument and never substitutes a value of its own, so "60" has exactly one source of truth in the repo.
• Tie-break is ascending (content_hash, chunk_index) (decision 3). Equal fused scores sort by that pair — content-derived, so it survives a re-ingest, and never dependent on dict or set iteration order. It is the same tie-break the stores themselves use (index/bm25.py, index/dense.py), extended by chunk_index because content_hash alone is not total: duplicate content shares a hash. chunk_id is deliberately not used despite being unique — it is uuid4 per ingest (contracts.py), so tied results would reorder every time the same corpus was re-indexed, breaking the across-runs determinism ADR-0005 decision 3 requires.
• No clamping (decision 5). With rrf_k > 0 (enforced below) and ranks starting at 1, every summed term 1 / (rrf_k + rank) is strictly positive by construction, so fused scores satisfy :class:~groundkit.contracts.RetrievalResult's ge=0.0 contract without a defensive max(0.0, ...) — a clamp here would mask a bug rather than prevent one.

This module builds (Chunk, score) pairs only, exactly as :meth:~groundkit.index.bm25.BM25Index.search and the dense stores' search do — never :class:~groundkit.contracts.RetrievalResult. The document-source join that turns a ranked chunk into a citation-bearing result belongs to retrieval/search.py (ADR-0006); fusion has no store to join against and performs no I/O.

reciprocal_rank_fusion

reciprocal_rank_fusion(
    rankings: Sequence[Sequence[tuple[Chunk, float]]],
    *,
    rrf_k: int,
    top_k: int | None = None,
) -> list[tuple[Chunk, float]]

Fuse multiple rankings of (Chunk, score) pairs by reciprocal rank.

Each input score is ignored — only the 1-based rank of a chunk within its own ranking contributes to the fused score (ADR-0005 decision 1). A chunk's identity across rankings is its chunk_id.

Parameters:

Name	Type	Description	Default
rankings	Sequence[Sequence[tuple[Chunk, float]]]	One sequence of (Chunk, score) pairs per retriever, each already sorted best-first. Input scores are ignored; only list position (rank) matters. Not mutated.	required
rrf_k	int	RRF's rank-constant. Must be > 0. Callers pass RetrievalConfig.rrf_k (default 60; provenance in ADR-0005 decision 2) — this function has no default of its own.	required
top_k	int | None	Maximum number of fused results to return, applied after sorting. None returns every fused chunk. Must be > 0 if given.	None

Returns:

Type	Description
list[tuple[Chunk, float]]	(Chunk, score) pairs sorted by descending fused score, ties
list[tuple[Chunk, float]]	broken by ascending (content_hash, chunk_index) (ADR-0005
list[tuple[Chunk, float]]	decision 3) — content-derived, so the ordering is identical across
list[tuple[Chunk, float]]	re-ingests of the same corpus. When a chunk_id appears in more than
list[tuple[Chunk, float]]	one ranking, the emitted Chunk object is the one from the first
list[tuple[Chunk, float]]	(lowest-index) ranking containing it. Empty rankings, or
list[tuple[Chunk, float]]	rankings that are all empty, yield [].

Raises:

Type	Description
ValueError	rrf_k <= 0; top_k is not None and top_k <= 0; or a single ranking contains the same chunk_id more than once (a caller bug — fail loud rather than silently double-counting it).

Source code in src/groundkit/retrieval/fusion.py

def reciprocal_rank_fusion(
    rankings: Sequence[Sequence[tuple[Chunk, float]]],
    *,
    rrf_k: int,
    top_k: int | None = None,
) -> list[tuple[Chunk, float]]:
    """Fuse multiple rankings of ``(Chunk, score)`` pairs by reciprocal rank.

    Each input score is ignored — only the 1-based rank of a chunk within
    its own ranking contributes to the fused score (ADR-0005 decision 1).
    A chunk's identity across rankings is its ``chunk_id``.

    Args:
        rankings: One sequence of ``(Chunk, score)`` pairs per retriever,
            each already sorted best-first. Input scores are ignored; only
            list position (rank) matters. Not mutated.
        rrf_k: RRF's rank-constant. Must be > 0. Callers pass
            ``RetrievalConfig.rrf_k`` (default 60; provenance in ADR-0005
            decision 2) — this function has no default of its own.
        top_k: Maximum number of fused results to return, applied after
            sorting. ``None`` returns every fused chunk. Must be > 0 if
            given.

    Returns:
        ``(Chunk, score)`` pairs sorted by descending fused score, ties
        broken by ascending ``(content_hash, chunk_index)`` (ADR-0005
        decision 3) — content-derived, so the ordering is identical across
        re-ingests of the same corpus. When a chunk_id appears in more than
        one ranking, the emitted ``Chunk`` object is the one from the first
        (lowest-index) ranking containing it. Empty ``rankings``, or
        rankings that are all empty, yield ``[]``.

    Raises:
        ValueError: ``rrf_k <= 0``; ``top_k`` is not ``None`` and
            ``top_k <= 0``; or a single ranking contains the same
            ``chunk_id`` more than once (a caller bug — fail loud rather
            than silently double-counting it).
    """
    if rrf_k <= 0:
        raise ValueError(f"rrf_k must be > 0, got {rrf_k}")
    if top_k is not None and top_k <= 0:
        raise ValueError(f"top_k must be > 0 if given, got {top_k}")

    fused_scores: dict[str, float] = {}
    first_chunk: dict[str, Chunk] = {}

    for ranking in rankings:
        seen_in_ranking: set[str] = set()
        for rank, (chunk, _score) in enumerate(ranking, start=1):
            chunk_id = chunk.chunk_id
            if chunk_id in seen_in_ranking:
                raise ValueError(f"duplicate chunk_id {chunk_id!r} within one ranking")
            seen_in_ranking.add(chunk_id)

            fused_scores[chunk_id] = fused_scores.get(chunk_id, 0.0) + 1.0 / (rrf_k + rank)
            if chunk_id not in first_chunk:
                first_chunk[chunk_id] = chunk

    fused = [(first_chunk[chunk_id], score) for chunk_id, score in fused_scores.items()]
    # Sort on content, not identity: chunk_id groups a chunk across rankings
    # (it is unique within one search) but is uuid4 per ingest, so using it
    # here would reorder tied results on every re-index. See ADR-0005
    # decision 3 and the module docstring.
    fused.sort(key=lambda pair: (-pair[1], pair[0].content_hash, pair[0].chunk_index))

    if top_k is not None:
        fused = fused[:top_k]

    return fused

Rerank

rerank_by_logits re-scores results but must not re-derive them: each reranked RetrievalResult carries forward the source_class and extractor of the result it replaces, rather than reverting to the contract's "text"/None defaults. That is what keeps citation verification (see Citations below) correct for a snapshot or extracted document after a rerank pass, not only before one.

groundkit.retrieval.rerank

Optional cross-encoder rerank using a local, non-LLM model (Phase 3, Wave D).

A cross-encoder scores each (query, passage) pair jointly rather than comparing two independently-computed vectors, which is why it is worth a second pass over an already-retrieved candidate list. It is not an LLM and not a generation step: SPEC.md §2's "deterministic core, LLM at the boundary" holds — this runs a small local sequence-classification model, in the retrieval path, on a fixed candidate set, and returns a permutation of what it was given.

Nothing here imports torch or sentence-transformers at module scope. The import is deferred to :func:_import_cross_encoder, called only when a :class:CrossEncoderReranker actually loads its model, so import groundkit.retrieval.rerank — and therefore every pure function below, and the whole default test suite — works in a base install. That matters twice over: retrieval/* is a glob in the core coverage subset (pyproject.toml), so this module is held to the core gate, and it could not be if exercising it required a multi-gigabyte optional dependency.

Hazard 2 is closed here, structurally

ADR-0001 hazard 2: ARP fed raw cross-encoder logits straight into a score >= 0.0 field and crashed, because MS MARCO-style rerankers emit unbounded logits and routinely emit negative ones. ADR-0005 decision 4 fixes the shape of the fix — a sigmoid, never min-max, never a clamp:

• Total. Sigmoid maps every real number into (0, 1), so no logit, however negative, can violate ge=0.0. The contract is satisfied by construction rather than by defending against the value after the fact.
• Monotonic. It cannot change the ranking, which is the only property a reranker owes. A clamp would satisfy the contract while collapsing every negatively-scored document into one tie — contract-legal and corrupt, the worst combination (ADR-0005, alternatives).
• Batch-independent. Min-max would make the same document score differently depending on what it was ranked alongside, and degenerates to a zero range on a single-result batch.

The squashing is done by :func:sigmoid in this module, over raw logits, rather than by asking sentence-transformers for pre-activated scores. ADR-0005 decision 4 requires the activation to be set explicitly and never inherited from a library default that could change; this goes one step further in the same direction, so that the total-function guarantee lives in groundkit's own code where :func:rerank_by_logits's regression test can exercise the exact path production uses. :func:_import_cross_encoder still sets the model's activation explicitly — to identity — so what arrives here is logits by declaration rather than by assumption, and a library default flipping cannot silently double-squash the scores.

CrossEncoderReranker

CrossEncoderReranker(
    model_name: str = DEFAULT_RERANK_MODEL,
    *,
    max_length: int | None = None,
)

Rerank retrieval results with a local cross-encoder.

Satisfies :class:~groundkit.retrieval.protocols.RerankerProtocol, whose signature is held by a conformance test (ADR-0001 hazard 4) — the parameter is query, and renaming it is the exact drift that test exists to catch.

The model is loaded lazily on first :meth:rerank and cached for the instance's lifetime. Construction is therefore cheap and total: it never touches the filesystem, the network, or the optional extra, so a misconfigured install fails at the first real call with :class:~groundkit.errors.RerankerNotConfiguredError rather than at import.

Usage::

reranker = CrossEncoderReranker()
top = await reranker.rerank("why is the sky blue", candidates, top_k=5)

Record the model identity; load nothing.

Parameters:

Name	Type	Description	Default
model_name	str	Cross-encoder identifier passed to sentence-transformers.	DEFAULT_RERANK_MODEL
max_length	int | None	Optional token truncation length for the pair encoder. None leaves the model's own configured length in place.	None

Source code in src/groundkit/retrieval/rerank.py

def __init__(
    self,
    model_name: str = DEFAULT_RERANK_MODEL,
    *,
    max_length: int | None = None,
) -> None:
    """Record the model identity; load nothing.

    Args:
        model_name: Cross-encoder identifier passed to sentence-transformers.
        max_length: Optional token truncation length for the pair encoder.
            ``None`` leaves the model's own configured length in place.
    """
    self._model_name = model_name
    self._max_length = max_length
    self._model: Any | None = None
    self._lock = asyncio.Lock()
    self._load_task: asyncio.Task[Any] | None = None

model_name property

model_name: str

The configured cross-encoder identifier.

rerank async

rerank(
    query: str,
    results: list[RetrievalResult],
    *,
    top_k: int = 5,
) -> list[RetrievalResult]

Return results reordered by cross-encoder relevance, capped at top_k.

An empty candidate list short-circuits before the model is loaded: there is nothing to reorder, and loading a multi-gigabyte model to return [] would make an empty upstream stage pay the reranker's entire cost.

Both blocking steps run in a worker thread: the first-call model load (:meth:_ensure_model, also serialized so concurrent cold calls build one model rather than several) and then predict itself. Each is synchronous and long enough to matter — the load costs a torch import and possibly a weight download, predict costs seconds on a real batch — and either one left on the event loop would stall every other coroutine, including, in Phase 4, every other in-flight request, along with the cancellations and timeouts meant to bound it.

Parameters:

Name	Type	Description	Default
query	str	The user query, scored jointly against each passage.	required
results	list[RetrievalResult]	Candidates to reorder. Not mutated.	required
top_k	int	Maximum number of results to return.	5

Returns:

Type	Description
list[RetrievalResult]	A new list of at most top_k results, best-first, with
list[RetrievalResult]	sigmoid-normalized scores.

Raises:

Type	Description
RerankerNotConfiguredError	The optional extra is missing or the model could not be loaded. Never a silent passthrough.
RetrievalError	top_k is not positive, the model returned a misaligned or non-finite batch, or inference itself failed — every backend exception is translated, so a caller handling this repo's error root is covered on the scoring path too.

Source code in src/groundkit/retrieval/rerank.py

async def rerank(
    self,
    query: str,
    results: list[RetrievalResult],
    *,
    top_k: int = 5,
) -> list[RetrievalResult]:
    """Return ``results`` reordered by cross-encoder relevance, capped at ``top_k``.

    An empty candidate list short-circuits before the model is loaded:
    there is nothing to reorder, and loading a multi-gigabyte model to
    return ``[]`` would make an empty upstream stage pay the reranker's
    entire cost.

    **Both** blocking steps run in a worker thread: the first-call model
    load (:meth:`_ensure_model`, also serialized so concurrent cold calls
    build one model rather than several) and then ``predict`` itself. Each
    is synchronous and long enough to matter — the load costs a torch
    import and possibly a weight download, ``predict`` costs seconds on a
    real batch — and either one left on the event loop would stall every
    other coroutine, including, in Phase 4, every other in-flight request,
    along with the cancellations and timeouts meant to bound it.

    Args:
        query: The user query, scored jointly against each passage.
        results: Candidates to reorder. Not mutated.
        top_k: Maximum number of results to return.

    Returns:
        A new list of at most ``top_k`` results, best-first, with
        sigmoid-normalized scores.

    Raises:
        RerankerNotConfiguredError: The optional extra is missing or the
            model could not be loaded. Never a silent passthrough.
        RetrievalError: ``top_k`` is not positive, the model returned a
            misaligned or non-finite batch, or inference itself failed —
            every backend exception is translated, so a caller handling
            this repo's error root is covered on the scoring path too.
    """
    if top_k <= 0:
        raise RetrievalError(f"rerank top_k must be > 0, got {top_k}")
    if not results:
        return []

    model = await self._ensure_model()
    pairs = [[query, result.content] for result in results]
    try:
        raw = await asyncio.to_thread(model.predict, pairs)
    except GroundkitError:
        raise
    except Exception as exc:
        # Inference fails in ways the backend owns and this repo does not
        # model — CUDA OOM, a tokenizer rejecting an input, a device that
        # went away. Left unwrapped they cross the seam as arbitrary
        # third-party exceptions, so a caller handling RetrievalError (in
        # Phase 4, the request boundary) is unprotected on the one path
        # most likely to fail under load, while every other failure in
        # this module already arrives typed.
        #
        # The exception's message is deliberately NOT interpolated, unlike
        # `_load_model`'s. A load failure cannot carry corpus text; an
        # inference failure can — a tokenizer error routinely quotes the
        # input it choked on, which here is the query and the passages.
        # The cause is chained instead, so a traceback still carries the
        # detail while the message this raises cannot itself leak content.
        raise RetrievalError(
            f"Cross-encoder {self._model_name!r} failed while scoring "
            f"{len(pairs)} query/passage pairs ({type(exc).__name__}); "
            "see the chained cause for the backend's own message"
        ) from exc
    logits = _coerce_logits(raw, model_name=self._model_name)
    logger.debug(
        "Reranked %d candidates to %d with %s",
        len(results),
        min(top_k, len(results)),
        self._model_name,
    )
    return rerank_by_logits(results, logits, top_k=top_k)

rerank_by_logits

rerank_by_logits(
    results: Sequence[RetrievalResult],
    logits: Sequence[float],
    *,
    top_k: int,
) -> list[RetrievalResult]

Reorder results by logits, sigmoid-normalized, truncated to top_k.

The pure half of reranking: no model, no I/O, no optional dependency. Every ordering and contract guarantee this module makes is made here, which is what lets the hazard-2 regression test drive the production path directly with hand-written negative logits.

Results are rebuilt through :class:~groundkit.contracts.RetrievalResult's constructor rather than copied with model_copy(update=...). That is deliberate and load-bearing: model_copy skips validation, so a score that violated ge=0.0 would sail through it and the hazard-2 test would be asserting nothing. Going through the constructor means the contract is genuinely re-checked on every reranked result.

Ties break on (source, start_offset) — a chunk's position in its document, which is content-derived and survives re-ingest. chunk_id would be the obvious key and is deliberately not used, for the reason ADR-0005 decision 3 gives for the fusion tie-break: it does not survive re-ingestion of the same corpus, so a tie would resolve differently between two runs over identical content.

Parameters:

Name	Type	Description	Default
results	Sequence[RetrievalResult]	Candidates to reorder. Not mutated; a new list of new objects is returned.	required
logits	Sequence[float]	Raw model scores, positionally aligned with results.	required
top_k	int	Maximum number of results to return. Must be > 0.	required

Returns:

Type	Description
list[RetrievalResult]	A new list, best-first, of at most top_k new
list[RetrievalResult]	class:~groundkit.contracts.RetrievalResult objects carrying
list[RetrievalResult]	sigmoid-normalized scores.

Raises:

Type	Description
RetrievalError	logits and results differ in length (a misaligned batch would attach every score to the wrong passage), top_k is not positive, or a logit is non-finite.

Source code in src/groundkit/retrieval/rerank.py

def rerank_by_logits(
    results: Sequence[RetrievalResult],
    logits: Sequence[float],
    *,
    top_k: int,
) -> list[RetrievalResult]:
    """Reorder ``results`` by ``logits``, sigmoid-normalized, truncated to ``top_k``.

    The pure half of reranking: no model, no I/O, no optional dependency. Every
    ordering and contract guarantee this module makes is made here, which is
    what lets the hazard-2 regression test drive the production path directly
    with hand-written negative logits.

    Results are rebuilt through :class:`~groundkit.contracts.RetrievalResult`'s
    constructor rather than copied with ``model_copy(update=...)``. That is
    deliberate and load-bearing: ``model_copy`` skips validation, so a score
    that violated ``ge=0.0`` would sail through it and the hazard-2 test would
    be asserting nothing. Going through the constructor means the contract is
    genuinely re-checked on every reranked result.

    Ties break on ``(source, start_offset)`` — a chunk's position in its
    document, which is content-derived and survives re-ingest. ``chunk_id``
    would be the obvious key and is deliberately not used, for the reason
    ADR-0005 decision 3 gives for the fusion tie-break: it does not survive
    re-ingestion of the same corpus, so a tie would resolve differently between
    two runs over identical content.

    Args:
        results: Candidates to reorder. Not mutated; a new list of new objects
            is returned.
        logits: Raw model scores, positionally aligned with ``results``.
        top_k: Maximum number of results to return. Must be > 0.

    Returns:
        A new list, best-first, of at most ``top_k`` new
        :class:`~groundkit.contracts.RetrievalResult` objects carrying
        sigmoid-normalized scores.

    Raises:
        RetrievalError: ``logits`` and ``results`` differ in length (a
            misaligned batch would attach every score to the wrong passage),
            ``top_k`` is not positive, or a logit is non-finite.
    """
    if top_k <= 0:
        raise RetrievalError(f"rerank top_k must be > 0, got {top_k}")
    if len(logits) != len(results):
        raise RetrievalError(
            f"Reranker returned {len(logits)} scores for {len(results)} results. A "
            "misaligned batch attaches every score to the wrong passage, which "
            "reorders the list confidently and wrongly rather than failing."
        )

    rescored = [
        RetrievalResult(
            content=result.content,
            score=sigmoid(logit),
            document_id=result.document_id,
            chunk_id=result.chunk_id,
            source=result.source,
            source_class=result.source_class,
            extractor=result.extractor,
            start_offset=result.start_offset,
            end_offset=result.end_offset,
            metadata=result.metadata,
        )
        for result, logit in zip(results, logits, strict=True)
    ]
    rescored.sort(key=lambda r: (-r.score, r.source, r.start_offset))
    return rescored[:top_k]

sigmoid

sigmoid(logit: float) -> float

Map an unbounded logit into [0.0, 1.0], monotonically and totally.

Computed in two branches so neither exp call can overflow: for a large positive logit exp(-x) underflows harmlessly toward zero, and for a large negative one the algebraically-equal exp(x) / (1 + exp(x)) form does the same. The naive single-branch spelling raises OverflowError somewhere past x ≈ -745, which would turn "a very confident negative score" — an ordinary reranker output — into a crash.

Mathematically the range is the open interval (0, 1). In float64 it is the closed one: past :data:SIGMOID_SATURATION_LOGIT the result rounds to exactly 0.0 or 1.0. Both still satisfy :class:~groundkit.contracts.RetrievalResult's ge=0.0 bound, so saturation costs ordering resolution between two equally-hopeless candidates and never contract validity.

Parameters:

Name	Type	Description	Default
logit	float	Any finite float. Non-finite input is a caller bug and is rejected rather than propagated into a score.	required

Returns:

Type	Description
float	A float in [0.0, 1.0].

Raises:

Type	Description
RetrievalError	logit is NaN or infinite. A NaN score compares false against everything and would silently scramble the sort; infinity is a model that has gone wrong, not a confident result.

Source code in src/groundkit/retrieval/rerank.py

def sigmoid(logit: float) -> float:
    """Map an unbounded logit into ``[0.0, 1.0]``, monotonically and totally.

    Computed in two branches so neither ``exp`` call can overflow: for a large
    positive logit ``exp(-x)`` underflows harmlessly toward zero, and for a
    large negative one the algebraically-equal ``exp(x) / (1 + exp(x))`` form
    does the same. The naive single-branch spelling raises ``OverflowError``
    somewhere past ``x ≈ -745``, which would turn "a very confident negative
    score" — an ordinary reranker output — into a crash.

    Mathematically the range is the open interval ``(0, 1)``. In float64 it is
    the closed one: past :data:`SIGMOID_SATURATION_LOGIT` the result rounds to
    exactly ``0.0`` or ``1.0``. Both still satisfy
    :class:`~groundkit.contracts.RetrievalResult`'s ``ge=0.0`` bound, so
    saturation costs ordering resolution between two equally-hopeless
    candidates and never contract validity.

    Args:
        logit: Any finite float. Non-finite input is a caller bug and is
            rejected rather than propagated into a score.

    Returns:
        A float in ``[0.0, 1.0]``.

    Raises:
        RetrievalError: ``logit`` is NaN or infinite. A NaN score compares
            false against everything and would silently scramble the sort;
            infinity is a model that has gone wrong, not a confident result.
    """
    if not math.isfinite(logit):
        raise RetrievalError(
            f"Reranker produced a non-finite score ({logit!r}). A NaN compares false "
            "against every value and would silently scramble the ranking; an infinity "
            "is a broken model, not a confident one. Refusing to build a result from it."
        )
    if logit >= 0.0:
        return 1.0 / (1.0 + math.exp(-logit))
    exp_logit = math.exp(logit)
    return exp_logit / (1.0 + exp_logit)

Citations

groundkit.retrieval.citations

Citation resolution: verify a retrieved span against its source (Phase 1).

A citation is verifiable, not vibes (SPEC.md §2): given the source file and character offsets, these helpers read the span back out of the source and compare it to what retrieval returned. Pure code, no LLM.

resolve_citation async

resolve_citation(
    citation: Citation,
    allowed_base_dir: Path,
    *,
    snapshot_dir: Path | None = None,
) -> str

Return the exact text span a citation points at, read from its source.

Dispatches on citation.source_class (ADR-0016), because "read the source and slice" is only correct for one of the three classes. It is correct for text precisely because a text loader makes Document.content the file's decoded bytes, so offsets into content are offsets into the file.

For the other two, a plain read-and-slice is wrong rather than merely incomplete:

• extracted — offsets index deterministic extractor output. Reading a PDF as UTF-8 raises; reading an HTML file returns markup the offsets were never measured against. Resolving one needs the same extractor that produced it, and re-running a different version silently shifts every offset after the first difference — so this looks the citation's recorded extractor identity up in :func:~groundkit.extraction.active_extractors (ADR-0016 decision 2) and refuses when it is absent. On a match the source is re-extracted through that exact extractor and the result takes the same offset/drift check text does — re-extract, slice, compare, per ADR-0016 decision 1's table. A build with neither the pdf nor the html extra installed has an empty registry, so every extracted citation refuses there — still the correct answer, now for a reason the registry reports rather than a constant nothing could ever populate.
• snapshot — source is a URL, and it never reaches :func:~groundkit.utils.path_safety.ensure_within_base: that function validates only that its input is non-empty and null-byte-free, then hands it to os.path.realpath, which resolves "https://example.com/x" as a relative path under the current directory rather than rejecting it as not-a-path. The class — not a URL-sniffing path helper — is what keeps that boundary sharp (test_source_class.py's test_a_url_passes_path_containment_which_is_why_class_is_checked_first pins the hazard this dispatch exists to route around). Resolution instead reads the local copy :class:~groundkit.ingestion.url_loader.UrlLoader wrote at ingest time (ADR-0016 decision 4) — never a re-fetch, since a re-fetch is a different observation at a different time and a mismatch against it could not distinguish a stale index from a changed server — from snapshot_dir/document_id (:mod:groundkit.snapshots), itself containment-checked against snapshot_dir before it is ever opened. A caller that omits snapshot_dir gets verdict="unresolvable" naming the parameter it needed to supply, not a crash.

Every failure here sets :attr:~groundkit.errors.RetrievalError.verdict explicitly, at the raise site that already knows it best, so service.tools.handle_fetch_chunk can classify the outcome by reading an attribute instead of pattern-matching the message (ADR-0016 decision 6). drifted means the source was read and no longer matches; unresolvable means it could not be checked at all — a distinction an extractor-identity mismatch, and a missing snapshot_dir or snapshot file, both fall on the unresolvable side of, because nothing was re-read to disagree with.

Parameters:

Name	Type	Description	Default
citation	Citation	The citation to resolve.	required
allowed_base_dir	Path	Containment root the citation's source must resolve within (same path-safety barrier as ingestion). Unused for a snapshot citation, which is contained against snapshot_dir instead.	required
snapshot_dir	Path | None	The per-collection snapshot containment root (:func:~groundkit.snapshots.snapshot_dir_for), required only to resolve a snapshot citation. None for every caller that never resolves one — keyword-only and defaulted so no existing caller breaks (spec §10.1).	None

Returns:

Type	Description
str	source_text[start_offset:end_offset].

Raises:

Type

Description

RetrievalError

The source escapes allowed_base_dir (or the snapshot escapes snapshot_dir), cannot be read, is not valid UTF-8, fails to re-extract under its recorded extractor, is shorter than the cited offsets (source changed since indexing — verdict="drifted"), or belongs to a source class or extractor identity this build cannot resolve, or a snapshot citation was given no snapshot_dir (verdict="unresolvable" in every other case).

Source code in src/groundkit/retrieval/citations.py

async def resolve_citation(
    citation: Citation, allowed_base_dir: Path, *, snapshot_dir: Path | None = None
) -> str:
    """Return the exact text span a citation points at, read from its source.

    Dispatches on ``citation.source_class`` (ADR-0016), because "read the
    source and slice" is only correct for one of the three classes. It is
    correct for ``text`` precisely because a text loader makes
    ``Document.content`` the file's decoded bytes, so offsets into content are
    offsets into the file.

    For the other two, a plain read-and-slice is wrong rather than merely
    incomplete:

    - ``extracted`` — offsets index deterministic extractor output. Reading a
      PDF as UTF-8 raises; reading an HTML file returns markup the offsets were
      never measured against. Resolving one needs the *same* extractor that
      produced it, and re-running a different version silently shifts every
      offset after the first difference — so this looks the citation's recorded
      extractor identity up in
      :func:`~groundkit.extraction.active_extractors` (ADR-0016 decision 2) and
      refuses when it is absent. On a match the source is re-extracted through
      that exact extractor and the result takes the same offset/drift check
      ``text`` does — re-extract, slice, compare, per ADR-0016 decision 1's
      table. A build with neither the ``pdf`` nor the ``html`` extra installed
      has an empty registry, so every ``extracted`` citation refuses there —
      still the correct answer, now for a reason the registry reports rather
      than a constant nothing could ever populate.
    - ``snapshot`` — ``source`` is a URL, and it never reaches
      :func:`~groundkit.utils.path_safety.ensure_within_base`: that function
      validates only that its input is non-empty and null-byte-free, then
      hands it to ``os.path.realpath``, which resolves
      ``"https://example.com/x"`` as a *relative path* under the current
      directory rather than rejecting it as not-a-path. The class — not a
      URL-sniffing path helper — is what keeps that boundary sharp
      (``test_source_class.py``'s
      ``test_a_url_passes_path_containment_which_is_why_class_is_checked_first``
      pins the hazard this dispatch exists to route around). Resolution
      instead reads the local copy
      :class:`~groundkit.ingestion.url_loader.UrlLoader` wrote at ingest time
      (ADR-0016 decision 4) — never a re-fetch, since a re-fetch is a
      different observation at a different time and a mismatch against it
      could not distinguish a stale index from a changed server — from
      ``snapshot_dir/document_id`` (:mod:`groundkit.snapshots`), itself
      containment-checked against ``snapshot_dir`` before it is ever opened.
      A caller that omits ``snapshot_dir`` gets ``verdict="unresolvable"``
      naming the parameter it needed to supply, not a crash.

    Every failure here sets :attr:`~groundkit.errors.RetrievalError.verdict`
    explicitly, at the raise site that already knows it best, so
    ``service.tools.handle_fetch_chunk`` can classify the outcome by reading
    an attribute instead of pattern-matching the message (ADR-0016 decision
    6). ``drifted`` means the source was read and no longer matches;
    ``unresolvable`` means it could not be checked at all — a distinction an
    extractor-identity mismatch, and a missing ``snapshot_dir`` or snapshot
    file, both fall on the ``unresolvable`` side of, because nothing was
    re-read to disagree with.

    Args:
        citation: The citation to resolve.
        allowed_base_dir: Containment root the citation's source must
            resolve within (same path-safety barrier as ingestion). Unused
            for a ``snapshot`` citation, which is contained against
            ``snapshot_dir`` instead.
        snapshot_dir: The per-collection snapshot containment root
            (:func:`~groundkit.snapshots.snapshot_dir_for`), required only to
            resolve a ``snapshot`` citation. ``None`` for every caller that
            never resolves one — keyword-only and defaulted so no existing
            caller breaks (spec §10.1).

    Returns:
        ``source_text[start_offset:end_offset]``.

    Raises:
        RetrievalError: The source escapes ``allowed_base_dir`` (or the
            snapshot escapes ``snapshot_dir``), cannot be read, is not valid
            UTF-8, fails to re-extract under its recorded extractor, is
            shorter than the cited offsets (source changed since indexing —
            ``verdict="drifted"``), or belongs to a source class or extractor
            identity this build cannot resolve, or a ``snapshot`` citation
            was given no ``snapshot_dir`` (``verdict="unresolvable"`` in every
            other case).
    """
    if citation.source_class == "snapshot":
        return await _resolve_snapshot(citation, snapshot_dir)

    # `text` and `extracted` share the containment check; they differ only in
    # how text is obtained from the (already-contained) path.
    try:
        path = ensure_within_base(citation.source, allowed_base_dir)
    except ValueError as exc:
        raise RetrievalError(str(exc), verdict="unresolvable") from exc

    if citation.source_class == "extracted":
        registry = extraction.active_extractors()
        extractor = registry.get(citation.extractor) if citation.extractor else None
        if extractor is None:
            raise RetrievalError(
                f"cannot resolve an 'extracted' citation for {citation.source!r}: "
                f"the extractor identity recorded at ingest ({citation.extractor!r}) "
                "is not active in this build "
                f"({sorted(registry) or 'none registered'}). "
                "Re-deriving these offsets needs the exact extractor that produced "
                "them; refused rather than guessing (ADR-0016 decision 2).",
                verdict="unresolvable",
            )
        try:
            text = await extractor.extract(path)
        except IngestionError as exc:
            # Not `drifted`: nothing was successfully re-derived to disagree
            # with what was indexed, so the check never got far enough to
            # compare (ADR-0016 decision 6). Symmetric with the OSError /
            # UnicodeDecodeError handling on the `text` branch below.
            raise RetrievalError(
                f"cannot resolve an 'extracted' citation for {citation.source!r}: "
                f"re-extraction with {extractor.identity} failed: {exc}",
                verdict="unresolvable",
            ) from exc
    else:  # "text"
        try:
            text = await asyncio.to_thread(path.read_text, "utf-8")
        except OSError as exc:
            raise RetrievalError(
                f"Cannot read cited source {citation.source!r}: {exc}", verdict="unresolvable"
            ) from exc
        except UnicodeDecodeError as exc:
            raise RetrievalError(
                f"Cited source {citation.source!r} is not valid UTF-8: {exc}",
                verdict="unresolvable",
            ) from exc

    return _slice_verified_span(text, citation)

verify_citation async

verify_citation(
    citation: Citation,
    expected_content: str,
    allowed_base_dir: Path,
    *,
    snapshot_dir: Path | None = None,
) -> bool

True when the cited span in the source equals expected_content.

Raises:

Type	Description
RetrievalError	As :func:resolve_citation.

Source code in src/groundkit/retrieval/citations.py

async def verify_citation(
    citation: Citation,
    expected_content: str,
    allowed_base_dir: Path,
    *,
    snapshot_dir: Path | None = None,
) -> bool:
    """True when the cited span in the source equals ``expected_content``.

    Raises:
        RetrievalError: As :func:`resolve_citation`.
    """
    return (
        await resolve_citation(citation, allowed_base_dir, snapshot_dir=snapshot_dir)
        == expected_content
    )