Skip to content

Architecture: agentic-tools

Package name: agentic-tools | Version: 0.1.0 | Python: 3.11+

Executive summary

agentic-tools is the shared utility layer for the tafreeman/agentic-runtime-platform monorepo. It provides a multi-provider LLM client abstraction, a disk-backed SHA-256 response cache, model probing and provider discovery, a model bakeoff runner, a benchmark and LLM-as-judge evaluation framework, a structured error taxonomy, and script bootstrap helpers.

The package is defined by the repo-root pyproject.toml (project name agentic-tools), which is also the uv workspace root; there is no tools/pyproject.toml. Members agentic-workflows-v2 and agentic-v2-eval declare agentic-tools as a workspace dependency.

The central design choice is a static facade + provider adapters + shared disk cache pattern. Callers never instantiate a client object; they call LLMClient.generate_text(model_name, prompt) as a static method. The method dispatches to one of several routing backends (local ONNX and Windows AI are distinct local backends) based on a name prefix in the model string. A 24-hour, SHA-256-keyed disk cache sits in front of all provider calls when enabled.

Dependency diagram

graph TD
    subgraph workspace["agentic-tools (tools/, defined by the repo-root pyproject.toml)"]
        core["tools/core\nconfig · errors · cache\nresponse_cache · prompt_db"]
        llm["tools/llm\nLLMClient · provider_adapters\nModelProbe · ModelInventory\nbakeoff · rank_models"]
        bench["tools/agents/benchmarks\nBenchmarkRunner · llm_evaluator\nevaluation_pipeline · registry"]
        core --> llm
        core --> bench
        llm --> bench
    end

    awv2["agentic-workflows-v2\n(runtime + server)"]
    eval["agentic-v2-eval\n(evaluation framework)"]

    workspace --> awv2
    workspace --> eval

    awv2 -.->|"langchain/model_builders (legacy local: builder)\nimports LLMClient"| llm
    awv2 -.->|"server/datasets reads\nbenchmark datasets + loader"| bench
    eval -.->|"adapters/llm_client wraps LLMClient"| llm
    eval -.->|"datasets bridges the\nbenchmark registry"| bench

Technology stack

Component Technology Notes
Language Python 3.11+ from __future__ import annotations throughout
Build backend hatchling pyproject.toml as single config source
Workspace manager uv Workspace root is the repo-root pyproject.toml; members: agentic-workflows-v2, agentic-v2-eval
HTTP (sync) urllib.request stdlib Ollama, Azure Foundry, Azure OpenAI adapters
HTTP (async) aiohttp Available for async provider backends
Data validation Pydantic v2 model_dump() / model_validate() — not .dict() / .parse_obj()
OpenAI SDK openai >=3.27.0,<4 OpenAI and Azure OpenAI providers
Anthropic SDK anthropic >=1.13.0,<2 Claude provider
Numeric numpy >=2.4.6,<3 Upper bound for semver safety
Optional: Google google-generativeai Gemini provider
Optional: ONNX onnxruntime-genai Local ONNX model inference
Optional: Windows AI .NET bridge + WinRT Phi Silica NPU (Copilot+ PC)
GitHub Models gh CLI subprocess Retired upstream 2026-07-30; the gh: adapter remains but calls now fail (ADR-016)
LangChain langchain-core >=1.6.9,<2 Thin adapter in langchain_adapter.py

The ranges above mirror the repo-root pyproject.toml; that file and uv.lock are the source of truth.

Package structure

pyproject.toml                        # Repo root: package definition (agentic-tools) + uv workspace root
tools/
├── __init__.py
├── llm/                              # LLM client and provider layer
│   ├── llm_client.py                 # LLMClient static facade — primary entry point
│   ├── provider_adapters.py          # All provider adapter functions
│   ├── langchain_adapter.py          # LangChain compatibility shim
│   ├── local_model.py                # ONNX model wrapper (onnxruntime-genai)
│   ├── local_models.py               # LOCAL_MODELS catalog (key → aigallery path)
│   ├── local_model_cli.py            # CLI for local model operations
│   ├── local_model_discovery.py      # Auto-detect from ~/.cache/aigallery
│   ├── model_probe.py                # ModelProbe class (slim facade + public API)
│   ├── model_registry.py             # Curated model registry constants
│   ├── check_provider_limits.py      # Provider rate-limit inspection CLI
│   ├── list_gemini.py                # Gemini model listing helper
│   ├── probe_config.py               # ProbeResult dataclass, constants, cache I/O, with_retry
│   ├── probe_providers.py            # Per-provider probe dispatch
│   ├── probe_providers_cloud.py      # Cloud probe logic (GitHub, OpenAI, Gemini, Claude, Azure)
│   ├── probe_providers_local.py      # Local probe logic (ONNX, Ollama, LM Studio, AITK)
│   ├── probe_discovery.py            # discover_all_models() — multi-provider orchestration
│   ├── probe_discovery_providers.py  # Underlying per-provider discovery helpers
│   ├── model_inventory.py            # build_inventory() — passive + active capability scan
│   ├── model_locks.py                # PID-based lock files for concurrent ONNX loading
│   ├── model_bakeoff.py              # Multi-model bakeoff runner
│   ├── bakeoff_tasks.py              # TaskSpec dataclass + TASKS list
│   ├── bakeoff_reporting.py          # _write_reports(), _recommend_alignment()
│   ├── rank_models.py                # Score-based model ranking from probe + limits data
│   ├── run_local_concurrency.py      # Local concurrency stress harness
│   ├── windows_ai.py                 # WindowsAIModel — .NET bridge to Phi Silica
│   └── windows_ai_bridge/            # C# .NET bridge project (PhiSilicaBridge.csproj)
├── core/                             # Shared utilities
│   ├── config.py                     # ModelConfig, PathConfig, Config dataclasses
│   ├── errors.py                     # ErrorCode StrEnum + classify_error()
│   ├── cache.py                      # Function-based cache API (delegates to ResponseCache)
│   ├── response_cache.py             # ResponseCache class: SHA-256, 24h TTL, disk-backed
│   ├── prompt_db.py                  # JSON-file-backed prompt/rubric/evaluation store
│   ├── _encoding.py                  # Windows console encoding fix (UTF-8)
│   ├── tool_init.py                  # ToolInit bootstrap dataclass
│   ├── local_media.py                # Local media file handling
│   └── model_availability.py        # Runtime model availability helpers
├── agents/
│   ├── repo_analyzer/                # Repository analysis agent helpers
│   └── benchmarks/                   # Benchmark infrastructure
│       ├── registry.py               # BenchmarkRegistry + PRESET_CONFIGS
│       ├── datasets.py               # BENCHMARK_DEFINITIONS per benchmark
│       ├── loader.py                 # load_benchmark() + clear_cache()
│       ├── runner.py                 # run_benchmark() + interactive CLI
│       ├── runner_commands.py        # cmd_* CLI handler functions
│       ├── runner_ui.py              # colorize, print_table, interactive_mode
│       ├── llm_evaluator.py          # LLM-as-judge: build_evaluation_prompt, evaluate_with_llm
│       ├── evaluator_models.py       # EvaluationResult, DimensionScore, SCORE_RUBRIC
│       ├── evaluator_reporting.py    # summarize_batch_results, save_evaluation_report
│       ├── evaluation_pipeline.py    # evaluate_task_output_llm, get_gold_standard_for_task
│       └── workflow_pipeline.py      # extract_workflow_data, save_workflow_phases_md
└── tests/                            # Key-free test suite
    ├── conftest.py
    ├── test_benchmark_pipeline.py
    ├── test_config.py
    ├── test_errors.py
    ├── test_evaluation_pipeline.py
    ├── test_llm_client.py
    ├── test_llm_evaluator.py
    ├── test_local_model_parsing.py
    ├── test_model_inventory.py
    ├── test_model_probe.py
    ├── test_model_registry_constants.py
    ├── test_probe_discovery_cloud.py
    ├── test_repo_analyzer.py
    ├── test_tool_init.py
    └── test_workflow_pipeline.py

LLM client

Core interface

from tools.llm.llm_client import LLMClient

# Synchronous, static — no instance required
response = LLMClient.generate_text(
    model_name="gh:gpt-4o-mini",
    prompt="Explain Kahn's algorithm.",
    system_instruction="You are a concise technical writer.",  # optional
    temperature=0.7,                                           # default
    max_tokens=4096,                                           # default
)

LLMClient is a pure static-method facade. No instance state exists. Thread safety is delegated to the ResponseCache layer, which uses a threading.Lock on all read/write operations.

Call flow

Caller
  └─ LLMClient.generate_text(model_name, prompt, ...)
       ├─ ResponseCache.get(sha256_key)         ← 24h disk cache
       │    └─ hit → return cached string
       └─ miss → provider_adapters.dispatch(prefix)
            ├─ local:*           → call_local()        (onnxruntime-genai)
            ├─ ollama:*          → call_ollama()        (urllib.request REST)
            ├─ windows-ai:*      → call_windows_ai()   (.NET bridge → Phi Silica)
            ├─ azure-foundry:*   → call_azure_foundry() (urllib.request REST)
            ├─ azure-openai:*    → call_azure_openai() (openai.AzureOpenAI SDK)
            ├─ gh:*              → call_github_models() (gh CLI subprocess)
            ├─ openai:*          → call_openai()        (openai.OpenAI SDK)
            ├─ gemini:*          → call_gemini()        (google.generativeai SDK)
            └─ claude:*          → call_claude()        (anthropic.Anthropic SDK)
                  └─ ResponseCache.set(sha256_key, response)

Provider routing table

Prefix Provider Transport Required Env Var(s)
local:* Local ONNX onnxruntime-genai in-process none; models in ~/.cache/aigallery
ollama:* Ollama REST http://localhost:11434 (default) OLLAMA_HOST (optional override)
windows-ai:* Windows Copilot Runtime / Phi Silica .NET bridge subprocess Win 11 Copilot+ PC + dotnet
azure-foundry:* Azure AI Foundry urllib.request REST AZURE_FOUNDRY_API_KEY, AZURE_FOUNDRY_ENDPOINT_1
azure-openai:* Azure OpenAI Service openai.AzureOpenAI SDK AZURE_OPENAI_ENDPOINT_0, AZURE_OPENAI_API_KEY_0
gh:* GitHub Models gh CLI subprocess GITHUB_TOKEN
openai:* OpenAI (hosted) openai.OpenAI SDK OPENAI_API_KEY
gemini:* Google Gemini google.generativeai SDK GEMINI_API_KEY or GOOGLE_API_KEY
claude:* Anthropic Claude anthropic.Anthropic SDK ANTHROPIC_API_KEY or CLAUDE_API_KEY
bare gpt* OpenAI (inferred) openai.OpenAI SDK OPENAI_API_KEY
bare gemini* Google Gemini (inferred) google.generativeai GEMINI_API_KEY
bare claude* Anthropic (inferred) anthropic.Anthropic ANTHROPIC_API_KEY

Remote provider gate

Remote providers (OpenAI, Anthropic, Gemini, Azure variants) are disabled by default. This prevents accidental API spend during local development and CI runs.

# Blocked unless PROMPTEVAL_ALLOW_REMOTE=1
LLMClient.generate_text("gpt-4o", "Hello")  # raises RuntimeError

# Exempt from this gate; GitHub still makes a remote, token-backed call
LLMClient.generate_text("local:phi4mini", "Hello")     # OK
LLMClient.generate_text("gh:gpt-4o-mini", "Hello")     # OK
LLMClient.generate_text("ollama:llama3", "Hello")       # OK
LLMClient.generate_text("windows-ai:phi-silica", "Hi") # OK

Set PROMPTEVAL_ALLOW_REMOTE=1 to enable the remote providers (OpenAI, Anthropic, Gemini, and the Azure variants). GitHub Models is exempt from this application gate, but it still sends a network request and its availability, quota, and billing depend on the current account.

Response cache

Property Value
Algorithm SHA-256 of model, system prompt and prompt content; temperature and max_tokens are folded into the prompt content by the caller
TTL 24 hours (configurable)
Storage ~/.cache/prompts-eval/responses/<first-2-hex>/<sha256_hex>.json plus cache_index.json
Max size MAX_CACHE_SIZE_MB = 500 is defined but not enforced; entries expire by TTL only (cleanup_expired())
Thread safety threading.Lock on all read/write operations
Enable PROMPTS_CACHE_ENABLED=1 (default)
Disable PROMPTS_CACHE_ENABLED=0

The cache is transparent to callers. When PROMPTS_CACHE_ENABLED is on, generate_text() consults it first and populates it after every successful response; the variable is re-read on each call.

Known defect

llm_client.py imports the cache as top-level modules (from cache import ...) and core/cache.py does from response_cache import .... Those names resolve only when tools/core is itself on sys.path. In a normal editable install the import raises ImportError, which generate_text() catches and treats as "caching unavailable", so the cache is silently skipped even though PROMPTS_CACHE_ENABLED defaults to on.

Model probing subsystem

ModelProbe provides runtime availability checking before evaluation runs. Rather than attempting an inference that may fail or incur cost, the probe sends a minimal test request and caches the result with TTL-stratified expiry.

ModelProbe
  ├─ check_model(model, force_probe=False) → ProbeResult
  ├─ filter_runnable(models) → list[str]
  ├─ get_probe_report(models) → dict
  ├─ clear_cache(model=None)
  └─ get_cache_summary() → dict

discover_all_models(verbose=False) → dict   # multi-provider scan
build_inventory(active_probes=False) → dict # passive + active capability inventory

TTL strategy: - Successful probe: 1 hour cache - Permanent error (403, model unavailable): 24 hours cache - Transient error (rate limit, timeout): 5 minutes cache

Model bakeoff

model_bakeoff.py runs a repeatable capability comparison across discovered models and recommends values for DEEP_RESEARCH_SMALL_MODEL and DEEP_RESEARCH_HEAVY_MODEL.

Scoring combines: - Task completion score (0–100, per-task weighted checks) - Latency penalty (min(elapsed_s, 120) × 0.15)

Output: JSON + Markdown reports written to reports/model-bakeoff by default (--out-dir overrides it).

Benchmark infrastructure

tools/agents/benchmarks/ provides an LLM-as-judge evaluation framework usable independently of the bakeoff. The primary entry point is evaluate_with_llm() in llm_evaluator.py, which scores output on five weighted dimensions (Completeness, Correctness, Quality, Specificity, Alignment) on a 0.0–10.0 scale.

Consumers define their own acceptance thresholds. The shared benchmark package does not impose a global coverage or source-quality gate.

Error taxonomy

tools.core.errors defines a StrEnum with ten values, two membership sets, and a heuristic classifier.

Code Meaning Retryable
SUCCESS No error —
UNAVAILABLE_MODEL Model not found or not loaded No
PERMISSION_DENIED Access denied (403/401) or remote gate No
RATE_LIMITED Provider rate limit exceeded (429) Yes
TIMEOUT Request exceeded timeout Yes
PARSE_ERROR Could not parse LLM response JSON Yes
FILE_NOT_FOUND Required file path missing No
INVALID_INPUT Input failed validation No
NETWORK_ERROR DNS/connection/network failure Yes
INTERNAL_ERROR Unexpected error No 
from tools.core.errors import classify_error, ErrorCode

code, should_retry = classify_error("Rate limit exceeded")
# → (ErrorCode.RATE_LIMITED, True)

Core configuration

from tools.core.config import default_config

# Loaded at import time from environment
default_config.models.generator_model   # GEN_MODEL env var, default "gpt-4o-mini"
default_config.models.reviewer_model    # REV_MODEL env var
default_config.models.refiner_model     # REF_MODEL env var

default_config.models.generator_temp    # 0.7
default_config.models.reviewer_temp     # 0.0
default_config.models.refiner_temp      # 0.5

Dependency installation

# Core (no optional providers) — run from the repo root, which owns the package
pip install -e .

# With Google Gemini support
pip install google-generativeai

# With local ONNX model inference
pip install onnxruntime-genai

# With Windows AI (Phi Silica) — Windows only
pip install winrt-runtime

# Dev dependencies (test, lint, typecheck)
pip install -e ".[dev]"

Environment variables reference

Variable Provider / Feature Notes
OPENAI_API_KEY OpenAI Required for openai:* and bare gpt*
ANTHROPIC_API_KEY Anthropic Claude Required for claude:*
GEMINI_API_KEY / GOOGLE_API_KEY Google Gemini Required for gemini:*
GITHUB_TOKEN / GH_TOKEN GitHub Models Required for gh:*
AZURE_OPENAI_ENDPOINT_0 Azure OpenAI Slot-indexed; _0 through _n for failover
AZURE_OPENAI_API_KEY_0 Azure OpenAI Paired with endpoint slot
AZURE_OPENAI_API_VERSION Azure OpenAI Default in this package: 2024-02-15-preview (the runtime's own backend defaults to 2024-10-21)
AZURE_FOUNDRY_API_KEY Azure Foundry Single key for all Foundry endpoints
AZURE_FOUNDRY_ENDPOINT_1 Azure Foundry Model slot 1 (phi4mini, phi4)
AZURE_FOUNDRY_ENDPOINT_2 Azure Foundry Model slot 2 (mistral)
OLLAMA_HOST Ollama Default: http://localhost:11434
OLLAMA_TIMEOUT_SECONDS Ollama Default: 180
LOCAL_MODEL_PATH Local ONNX Override; auto-detected from ~/.cache/aigallery
PROMPTS_CACHE_ENABLED Response cache 1 (default) / 0 to disable
PROMPTEVAL_ALLOW_REMOTE Remote gate 1 to enable OpenAI/Claude/Gemini/Azure
GEN_MODEL / REV_MODEL / REF_MODEL Config Override default pipeline model assignments
PHI_SILICA_LAF_FEATURE_ID Windows AI Limited Access Feature token (if required)
PHI_SILICA_LAF_TOKEN Windows AI LAF unlock token

Testing

All tests in tools/tests/ are fully mocked — no live LLM calls are made. The suite covers LLMClient, ModelProbe, ModelInventory, llm_evaluator, evaluation_pipeline, and config.

# From repo root
python -m pytest tools/tests/ -v

# With coverage
python -m pytest tools/tests/ -q --cov=tools --cov-report=term-missing

Markers: slow, integration. Skip with pytest -m 'not integration'.

See tools/tests/test_llm_client.py for representative patterns using unittest.mock.patch.