Eval harness

Most of the harness is reached through grk eval — the eval harness guide covers what it measures, what it deliberately does not, and how to reproduce any report it writes.

The two modules below have no CLI verb and are called directly from Python. That is deliberate in both cases: one writes a corpus, which the read-only eval surface is not the place for, and the other answers a question about a report after the fact rather than during a run.

Significance

significant is decided by the percentile confidence interval and never by p_value_two_sided; the two are computed independently. That p-value is a Monte Carlo achieved significance level over the observed deltas rather than a null-centred p-value, and it carries a (r + 1) / (B + 1) finite-sample correction — so it has a floor and can never be reported as exactly zero.

The unit of resampling is a query. Pairing is checked by query id, so a missing or reordered query raises rather than silently becoming an unpaired comparison, and the generator is seeded per call so a result does not depend on call order.

groundkit.evals.significance

Deterministic paired-bootstrap comparisons for retrieval experiments.

The unit of resampling is a query, not an individual retrieved hit. Callers must therefore provide one scalar score per query for both systems. Pairing by query id is checked explicitly so a missing or reordered query cannot turn into an unpaired comparison by accident.

BootstrapResult

Bases: BaseModel

Machine-readable result of one paired-bootstrap comparison.

compare_report_stages

compare_report_stages(
    report: EvalReport,
    *,
    baseline: StageName,
    candidate: StageName,
    metric: MetricName,
    resamples: int = 10000,
    seed: int = 7,
    confidence_level: float = 0.95,
) -> BootstrapResult

Compare two stages from the same eval artifact.

Source code in src/groundkit/evals/significance.py

def compare_report_stages(
    report: EvalReport,
    *,
    baseline: StageName,
    candidate: StageName,
    metric: MetricName,
    resamples: int = 10_000,
    seed: int = 7,
    confidence_level: float = 0.95,
) -> BootstrapResult:
    """Compare two stages from the same eval artifact."""

    # `EvalReport`'s validator pins exactly one baseline, at position zero,
    # named "bm25" -- but nothing stops two later stages sharing a name. A
    # dict comprehension would keep the last silently, so which `dense` stage
    # a comparison actually measured would depend on their order in the file.
    duplicate_stages = sorted(
        {stage.stage for stage in report.stages if _count_stage(report, stage.stage) > 1}
    )
    if duplicate_stages:
        raise EvalError(
            f"report contains more than one stage named {duplicate_stages}; a comparison "
            "cannot say which of them it measured"
        )
    stages = {stage.stage: stage for stage in report.stages}
    try:
        baseline_stage = stages[baseline]
        candidate_stage = stages[candidate]
    except KeyError as exc:
        raise EvalError(f"report does not contain stage {exc.args[0]!r}") from exc

    baseline_scores = _query_scores(baseline_stage.queries, metric)
    candidate_scores = _query_scores(candidate_stage.queries, metric)
    return paired_bootstrap(
        baseline_scores,
        candidate_scores,
        baseline_name=baseline,
        candidate_name=candidate,
        metric=metric,
        resamples=resamples,
        seed=seed,
        confidence_level=confidence_level,
    )

paired_bootstrap

paired_bootstrap(
    baseline_scores: Mapping[str, float],
    candidate_scores: Mapping[str, float],
    *,
    baseline_name: str,
    candidate_name: str,
    metric: MetricName,
    resamples: int = 10000,
    seed: int = 7,
    confidence_level: float = 0.95,
) -> BootstrapResult

Compare two systems by resampling paired query-level score deltas.

The confidence interval uses linear interpolation between ordered bootstrap means, and is what :attr:BootstrapResult.significant reads.

:attr:BootstrapResult.p_value_two_sided is twice the smaller empirical tail proportion around zero, carrying a (r + 1) / (B + 1) finite-sample correction so it is never reported as exactly zero. It is an achieved significance level computed from resamples of the observed deltas, not from a null-centred distribution -- see that field's own note. The two are computed independently: correcting one cannot move the other.

Source code in src/groundkit/evals/significance.py

def paired_bootstrap(
    baseline_scores: Mapping[str, float],
    candidate_scores: Mapping[str, float],
    *,
    baseline_name: str,
    candidate_name: str,
    metric: MetricName,
    resamples: int = 10_000,
    seed: int = 7,
    confidence_level: float = 0.95,
) -> BootstrapResult:
    """Compare two systems by resampling paired query-level score deltas.

    The confidence interval uses linear interpolation between ordered
    bootstrap means, and is what :attr:`BootstrapResult.significant` reads.

    :attr:`BootstrapResult.p_value_two_sided` is twice the smaller empirical
    tail proportion around zero, carrying a ``(r + 1) / (B + 1)``
    finite-sample correction so it is never reported as exactly zero. It is
    an achieved significance level computed from resamples of the *observed*
    deltas, not from a null-centred distribution -- see that field's own note.
    The two are computed independently: correcting one cannot move the other.
    """

    if resamples <= 0:
        raise EvalError(f"resamples must be positive, got {resamples}")
    if not 0.0 < confidence_level < 1.0:
        raise EvalError(f"confidence_level must be between 0 and 1, got {confidence_level}")
    if not baseline_scores:
        raise EvalError("paired bootstrap requires at least one query score")
    if baseline_scores.keys() != candidate_scores.keys():
        missing_candidate = sorted(baseline_scores.keys() - candidate_scores.keys())
        missing_baseline = sorted(candidate_scores.keys() - baseline_scores.keys())
        raise EvalError(
            "paired bootstrap query ids differ "
            f"(missing from candidate={missing_candidate}, "
            f"missing from baseline={missing_baseline})"
        )

    query_ids = sorted(baseline_scores)
    deltas = [candidate_scores[qid] - baseline_scores[qid] for qid in query_ids]
    if not all(math.isfinite(delta) for delta in deltas):
        raise EvalError("paired bootstrap scores must be finite")

    # A seeded simulation needs reproducibility, not cryptographic randomness.
    rng = random.Random(seed)  # noqa: S311
    query_count = len(deltas)
    bootstrap_means = [
        sum(deltas[rng.randrange(query_count)] for _ in range(query_count)) / query_count
        for _ in range(resamples)
    ]
    bootstrap_means.sort()

    alpha = (1.0 - confidence_level) / 2.0
    low = _quantile(bootstrap_means, alpha)
    high = _quantile(bootstrap_means, 1.0 - alpha)
    # Finite-sample correction, (r + 1) / (B + 1) rather than r / B. A Monte
    # Carlo estimate over B draws cannot resolve a probability below 1/B, so a
    # zero tail count means "smaller than this simulation can measure", never
    # "impossible". Reporting the uncorrected 0.0 states a certainty no finite
    # resampling can support, and it is reachable in practice, not just in
    # theory: any effect large and consistent enough that no resample crosses
    # zero produces it -- which is exactly what the strongest real contrasts do.
    non_positive = sum(value <= 0.0 for value in bootstrap_means)
    non_negative = sum(value >= 0.0 for value in bootstrap_means)
    p_value = min(1.0, 2.0 * (min(non_positive, non_negative) + 1) / (resamples + 1))

    return BootstrapResult(
        baseline=baseline_name,
        candidate=candidate_name,
        metric=metric,
        query_count=query_count,
        resamples=resamples,
        seed=seed,
        confidence_level=confidence_level,
        effect=sum(deltas) / query_count,
        confidence_interval_low=low,
        confidence_interval_high=high,
        p_value_two_sided=p_value,
        significant=low > 0.0 or high < 0.0,
    )

BEIR adaptation

BEIR relevance is per document and a groundkit judgment is a verbatim quote, so each adapted gold quote is the whole document text — document-level relevance expressed in the span vocabulary, rather than span annotations BEIR does not provide. A score computed over the adapted set is therefore not comparable to a published BEIR number, which ranks documents where grk eval ranks chunks; the guide's warning on that point states what comparing them would require.

A BEIR corpus is an untrusted third-party download, and every identifier that becomes part of a path is treated as such — validated as a single path component, containment-checked independently, and rejected before anything is written rather than after.

groundkit.evals.beir

Adapt BEIR datasets to groundkit's quote-anchored evaluation format.

Placed in :mod:groundkit.evals rather than a top-level datasets package because it imports :mod:groundkit.evals.corpus to reuse the harness's own judgment loader. GK-021 (tests/test_deterministic_core.py) bars every module outside groundkit.evals from importing the harness, so that the harness stays droppable from a plain library install; an adapter that exists only to feed the harness belongs inside that boundary rather than being an exception to it.

Validation order is the load-bearing property here. Every identifier is checked -- against both this module's path-safety class and the harness's own stricter contract -- before a single byte is written, so a dataset the harness would later reject leaves nothing behind on disk. An earlier shape of this module wrote the corpus first and re-validated afterwards, which turned a rejected dataset into a half-populated output directory.

BeirAdaptationReport

Bases: BaseModel

Counts and paths produced by :func:adapt_beir_dataset.

adapt_beir_dataset

adapt_beir_dataset(
    source_dir: Path,
    output_dir: Path,
    *,
    split: str = "test",
) -> BeirAdaptationReport

Convert a BEIR directory without downloading or filtering its data.

BEIR qrels are document-level. Each emitted gold quote is therefore the complete adapted document text. groundkit resolves the quote at offset zero and treats every overlapping chunk as relevant, preserving the qrel meaning without pretending that BEIR provides span-level annotations.

The source directory is untrusted input -- a BEIR corpus is a third-party download, and a mirror or hand-off can supply any _id it likes -- so every identifier is validated as a path component before it reaches a path join, and every resulting path is containment-checked against output_dir as a second, independent barrier.

Raises:

Type	Description
EvalError	The source files are missing or malformed, an identifier is unsafe as a path component or violates the harness's own contract, the qrels reference an unknown document or query, the split contains no positive relevance rows, or the output corpus directory already holds files (which would silently mix two datasets -- remove it and re-run).

Source code in src/groundkit/evals/beir.py

def adapt_beir_dataset(
    source_dir: Path, output_dir: Path, *, split: str = "test"
) -> BeirAdaptationReport:
    """Convert a BEIR directory without downloading or filtering its data.

    BEIR qrels are document-level.  Each emitted gold quote is therefore the
    complete adapted document text.  groundkit resolves the quote at offset
    zero and treats every overlapping chunk as relevant, preserving the qrel
    meaning without pretending that BEIR provides span-level annotations.

    The source directory is untrusted input -- a BEIR corpus is a third-party
    download, and a mirror or hand-off can supply any ``_id`` it likes -- so
    every identifier is validated as a path component before it reaches a
    path join, and every resulting path is containment-checked against
    ``output_dir`` as a second, independent barrier.

    Raises:
        EvalError: The source files are missing or malformed, an identifier is
            unsafe as a path component or violates the harness's own contract,
            the qrels reference an unknown document or query, the split
            contains no positive relevance rows, or the output corpus
            directory already holds files (which would silently mix two
            datasets -- remove it and re-run).
    """

    if not _SAFE_SPLIT.fullmatch(split):
        raise EvalError(
            f"unsafe BEIR split {split!r}; expected letters, digits, dot, underscore, or hyphen"
        )

    source = source_dir.resolve()
    destination = output_dir.resolve()
    corpus_records = _load_jsonl(source / "corpus.jsonl")
    query_records = _load_jsonl(source / "queries.jsonl")
    qrels = _load_qrels(source / "qrels" / f"{split}.tsv")

    texts: dict[str, str] = {}
    for position, record in enumerate(corpus_records, start=1):
        document_id = _required_string(record, "_id", context=f"corpus row {position}")
        _validate_identifier(document_id, subject="document", becomes_a_filename=True)
        if document_id in texts:
            raise EvalError(f"duplicate BEIR document id {document_id!r}")
        title = _optional_string(record, "title", context=f"corpus row {position}").strip()
        body = _optional_string(record, "text", context=f"corpus row {position}").strip()
        text = f"{title}{_TITLE_SEPARATOR}{body}" if title else body
        if not text:
            raise EvalError(f"BEIR document {document_id!r} has no title or text")
        texts[document_id] = text

    queries: dict[str, str] = {}
    for position, record in enumerate(query_records, start=1):
        query_id = _required_string(record, "_id", context=f"query row {position}")
        _validate_identifier(query_id, subject="query", becomes_a_filename=False)
        if query_id in queries:
            raise EvalError(f"duplicate BEIR query id {query_id!r}")
        query = _required_string(record, "text", context=f"query row {position}").strip()
        if not query:
            raise EvalError(f"BEIR query {query_id!r} is empty")
        queries[query_id] = query

    missing_documents = sorted(
        {doc_id for doc_ids in qrels.values() for doc_id in doc_ids} - texts.keys()
    )
    missing_queries = sorted(qrels.keys() - queries.keys())
    if missing_documents:
        raise EvalError(f"BEIR qrels reference missing documents: {missing_documents}")
    if missing_queries:
        raise EvalError(f"BEIR qrels reference missing queries: {missing_queries}")
    if not qrels:
        raise EvalError(f"BEIR qrels for split {split!r} contain no positive relevance rows")

    # The harness's query-id contract is stricter than `_SAFE_IDENTIFIER`
    # (kebab-case lowercase, so `MED-10` and `q_1` are path-safe but still
    # inadmissible). Checked here, before any write, so `load_judgments`
    # below can only fail on something this loop could not have known.
    unusable = sorted(qid for qid in qrels if not _HARNESS_QUERY_ID.fullmatch(qid))
    if unusable:
        raise EvalError(
            f"BEIR query ids are not admissible as groundkit judgment ids: {unusable}. "
            f"The harness requires kebab-case lowercase ({_QUERY_ID_PATTERN}); rename or "
            "map these ids before adapting, rather than writing a corpus the harness "
            "would reject on load."
        )

    _reject_case_colliding_document_ids(texts)
    _require_directory_or_absent(destination, "output")
    # Checked here, beside the other destination checks and *before* any
    # document is written. `judgments.jsonl` is opened only after the whole
    # corpus has landed, so a directory sitting at that path raised
    # IsADirectoryError with the output already populated -- a traceback and a
    # half-written dataset, from the one module whose contract is that a
    # rejected input leaves nothing behind.
    _require_absent_or_file(destination / "judgments.jsonl", "judgments")
    # Resolved *here* too, not at the open() below. Containment was the last
    # check standing between a symlinked judgments path and the filesystem, and
    # it ran after the corpus loop -- so the escape it caught, it caught with
    # every document already on disk, and reported as a raw ValueError rather
    # than an EvalError. Both halves of that are wrong for a module whose
    # contract is that a rejected input writes nothing. Binding the path now
    # makes the ordering structural instead of resting on the argument that a
    # literal filename cannot escape.
    judgments_path = _contained(destination / "judgments.jsonl", destination, "judgments")

    corpus_dir = destination / "corpus"
    _require_empty_corpus_dir(corpus_dir)
    corpus_dir.mkdir(parents=True, exist_ok=True)
    for document_id, text in sorted(texts.items()):
        # `_validate_identifier` already makes an escape unrepresentable; the
        # containment check is the independent second barrier the rest of this
        # package applies wherever a path is built from outside input, and the
        # one static analysis recognizes as a path-injection sanitizer.
        # Contained against `destination`, not `corpus_dir`. Checking against
        # `corpus_dir` resolves the candidate *and* the base through the same
        # link, so a `corpus` symlink pointing outside the destination passes
        # trivially and every document lands wherever it points.
        # `_require_empty_corpus_dir` refuses such a link outright; this is the
        # independent second barrier, and the one that stays correct if that
        # check is ever relaxed.
        filename = f"{document_id}{_DOCUMENT_SUFFIX}"
        document_path = _contained(corpus_dir / filename, destination, "document")
        document_path.write_text(text, encoding="utf-8", newline="\n")

    with judgments_path.open("w", encoding="utf-8", newline="\n") as handle:
        for query_id in sorted(qrels):
            row = {
                "query_id": query_id,
                "query": queries[query_id],
                "category": "normal",
                "gold": [
                    {"doc": f"{document_id}{_DOCUMENT_SUFFIX}", "quote": texts[document_id]}
                    for document_id in sorted(qrels[query_id])
                ],
            }
            handle.write(json.dumps(row, ensure_ascii=False) + "\n")

    # Reuse the harness's authoritative schema/path/order checks on our output.
    load_judgments(judgments_path)
    return BeirAdaptationReport(
        source_dir=str(source),
        output_dir=str(destination),
        split=split,
        document_count=len(texts),
        query_count=len(qrels),
        relevance_pair_count=sum(len(document_ids) for document_ids in qrels.values()),
        total_characters=sum(len(text) for text in texts.values()),
        corpus_dir=str(corpus_dir),
        judgments_path=str(judgments_path),
    )