Evaluation API

Single-run cases

executionkit.evals.EvalCase dataclass

EvalCase(name: str, run: EvalRun, check: EvalCheck, metadata: Mapping[str, Any] = dict())

A single eval case with a runner and a result check.

executionkit.evals.EvalResult dataclass

EvalResult(name: str, passed: bool, reason: str = '', metadata: MappingProxyType[str, Any] = (lambda: MappingProxyType({}))())

Outcome of a single eval case.

executionkit.evals.EvalReport dataclass

EvalReport(results: tuple[EvalResult, ...], min_accuracy: float | None = None)

Aggregate eval report.

Attributes:

Name	Type	Description
results	tuple[EvalResult, ...]	Per-case outcomes.
min_accuracy	float | None	Optional accuracy threshold used by :func:run_eval_suite for live / non-deterministic suites. When set, callers gate on :attr:accuracy_passed instead of :attr:passed to allow a small number of tolerated failures.

accuracy property

accuracy: float

Fraction of cases that passed; 0.0 when the suite is empty.

accuracy_passed property

accuracy_passed: bool

True when accuracy >= min_accuracy (live-suite gate).

Falls back to :attr:passed when no min_accuracy was configured, so callers can always use this property as the single gate regardless of suite type.

passed property

passed: bool

True when every case passed (100% — required for deterministic suites).

summary

summary() -> str

One-line human-readable result, e.g. '7/9 passed (77.8% accuracy)'.

Source code in executionkit/evals.py

def summary(self) -> str:
    """One-line human-readable result, e.g. '7/9 passed (77.8% accuracy)'."""
    return f"{self.passed_count}/{self.total} passed ({self.accuracy:.1%} accuracy)"

executionkit.evals.run_eval_suite async

run_eval_suite(cases: Sequence[EvalCase], *, min_accuracy: float | None = None) -> EvalReport

Run eval cases in order and return pass/fail results.

Parameters:

Name	Type	Description	Default
cases	Sequence[EvalCase]	Eval cases to run.	required
min_accuracy	float | None	When provided, the report is considered passing when report.accuracy >= min_accuracy rather than requiring every case to pass (report.passed). Use :data:LIVE_EVAL_MIN_ACCURACY for non-deterministic / live-provider suites that tolerate a small number of failures. Deterministic golden suites should omit this parameter so they continue to require 100% pass rate via report.passed.	None

Returns:

Type	Description
EvalReport	class:EvalReport with per-case results. The caller gates on either
EvalReport	report.passed (deterministic) or
EvalReport	report.accuracy >= min_accuracy (live).

Source code in executionkit/evals.py

async def run_eval_suite(
    cases: Sequence[EvalCase],
    *,
    min_accuracy: float | None = None,
) -> EvalReport:
    """Run eval cases in order and return pass/fail results.

    Args:
        cases: Eval cases to run.
        min_accuracy: When provided, the report is considered passing when
            ``report.accuracy >= min_accuracy`` rather than requiring every
            case to pass (``report.passed``).  Use :data:`LIVE_EVAL_MIN_ACCURACY`
            for non-deterministic / live-provider suites that tolerate a small
            number of failures.  Deterministic golden suites should omit this
            parameter so they continue to require 100% pass rate via
            ``report.passed``.

    Returns:
        :class:`EvalReport` with per-case results.  The caller gates on either
        ``report.passed`` (deterministic) or
        ``report.accuracy >= min_accuracy`` (live).
    """
    return EvalReport(
        results=tuple([await _run_case(case) for case in cases]),
        min_accuracy=min_accuracy,
    )

Conversation scripts

executionkit.evals.Turn dataclass

Turn(user: str, check: EvalCheck)

One turn of a multi-turn :class:ConversationScript.

Attributes:

Name	Type	Description
user	str	The user text sent to :meth:Kit.turn for this turn.
check	EvalCheck	An :data:EvalCheck applied to the PatternResult[str] that kit.turn returns. Returning None/True/"" passes the turn; returning False or a non-empty string fails it (the string becomes the failure reason).

executionkit.evals.ConversationScript dataclass

ConversationScript(name: str, turns: tuple[Turn, ...])

A named, ordered sequence of conversational turns to evaluate.

Run via :func:run_conversation_script, which drives each turn through a single :class:~executionkit.kit.Kit so conversation state carries across turns.

Attributes:

Name	Type	Description
name	str	Identifier used to label each turn's :class:EvalResult (f"{name}[{i}]" for turn i).
turns	tuple[Turn, ...]	The turns to run, in order.

executionkit.evals.run_conversation_script async

run_conversation_script(script: ConversationScript, kit: Kit) -> EvalReport

Drive a multi-turn :class:ConversationScript through a single Kit.

Each turn is sent via kit.turn(turn.user); because the same kit carries conversation state across calls, later turns see the transcript of earlier ones. The :class:PatternResult returned by each turn is passed to that turn's :data:EvalCheck, and the verdict is normalized exactly like :func:run_eval_suite does (None/True/"" pass; False and other strings fail). Any exception raised by kit.turn or the check is captured as a failed :class:EvalResult rather than propagated, so one bad turn does not abort the remaining turns.

Parameters:

Name	Type	Description	Default
script	ConversationScript	The named sequence of turns to evaluate.	required
kit	Kit	A :class:~executionkit.kit.Kit session whose provider satisfies :class:~executionkit.provider.ToolCallingProvider. State accrues on this kit as the script runs.	required

Returns:

Name	Type	Description
An	EvalReport	class:EvalReport with one :class:EvalResult per turn, each named
	EvalReport	f"{script.name}[{i}]". No min_accuracy is set, so the report
	EvalReport	gates on a 100% pass rate via :attr:EvalReport.passed.

Source code in executionkit/evals.py

async def run_conversation_script(
    script: ConversationScript,
    kit: Kit,
) -> EvalReport:
    """Drive a multi-turn :class:`ConversationScript` through a single ``Kit``.

    Each turn is sent via ``kit.turn(turn.user)``; because the same *kit*
    carries conversation state across calls, later turns see the transcript of
    earlier ones. The :class:`PatternResult` returned by each ``turn`` is passed
    to that turn's :data:`EvalCheck`, and the verdict is normalized exactly like
    :func:`run_eval_suite` does (``None``/``True``/``""`` pass; ``False`` and
    other strings fail). Any exception raised by ``kit.turn`` or the check is
    captured as a failed :class:`EvalResult` rather than propagated, so one bad
    turn does not abort the remaining turns.

    Args:
        script: The named sequence of turns to evaluate.
        kit: A :class:`~executionkit.kit.Kit` session whose provider satisfies
            :class:`~executionkit.provider.ToolCallingProvider`. State accrues
            on this kit as the script runs.

    Returns:
        An :class:`EvalReport` with one :class:`EvalResult` per turn, each named
        ``f"{script.name}[{i}]"``. No ``min_accuracy`` is set, so the report
        gates on a 100% pass rate via :attr:`EvalReport.passed`.
    """
    results: list[EvalResult] = []
    empty_metadata: MappingProxyType[str, Any] = MappingProxyType({})
    for index, turn in enumerate(script.turns):
        name = f"{script.name}[{index}]"
        try:
            result: PatternResult[str] = await kit.turn(turn.user)
            verdict = turn.check(result)
        except Exception as exc:
            results.append(
                EvalResult(
                    name=name,
                    passed=False,
                    reason=f"{type(exc).__name__}: {exc}",
                    metadata=empty_metadata,
                )
            )
            continue
        results.append(_verdict_to_result(name, verdict, empty_metadata))
    return EvalReport(results=tuple(results))

Live-provider helper

executionkit.evals.live_provider_from_env

live_provider_from_env() -> Provider | None

Build an opt-in live eval provider from EXECUTIONKIT_* env vars.

Returns None unless EXECUTIONKIT_LIVE_EVAL=1. When enabled, EXECUTIONKIT_BASE_URL and EXECUTIONKIT_MODEL are required; EXECUTIONKIT_API_KEY is optional and defaults to an empty string.

Source code in executionkit/evals.py

def live_provider_from_env() -> Provider | None:
    """Build an opt-in live eval provider from EXECUTIONKIT_* env vars.

    Returns ``None`` unless ``EXECUTIONKIT_LIVE_EVAL=1``. When enabled,
    ``EXECUTIONKIT_BASE_URL`` and ``EXECUTIONKIT_MODEL`` are required;
    ``EXECUTIONKIT_API_KEY`` is optional and defaults to an empty string.
    """

    if os.getenv("EXECUTIONKIT_LIVE_EVAL") != "1":
        return None

    base_url = os.getenv("EXECUTIONKIT_BASE_URL")
    model = os.getenv("EXECUTIONKIT_MODEL")
    missing = [
        name
        for name, value in (
            ("EXECUTIONKIT_BASE_URL", base_url),
            ("EXECUTIONKIT_MODEL", model),
        )
        if not value
    ]
    if missing:
        raise ValueError(f"{', '.join(missing)} required for live evals")

    return Provider(
        base_url=base_url or "",
        model=model or "",
        api_key=os.getenv("EXECUTIONKIT_API_KEY", ""),
    )

live_provider_from_env() reads:

Variable	Required when enabled
EXECUTIONKIT_LIVE_EVAL=1	Enables construction.
EXECUTIONKIT_BASE_URL	Yes
EXECUTIONKIT_MODEL	Yes
EXECUTIONKIT_API_KEY	No

It returns None when live evaluation is not enabled. It raises ValueError when the flag is enabled but a required value is missing.

Read the Evaluation guide for examples and the difference between repeatable and live checks.