Telemetry¶
OpenTelemetry spans and structured JSON logging (ADR-0022). A deliberate
leaf module: it imports nothing from groundkit, because
retrieval.search imports this module to instrument its search span, so
an import in the other direction would be circular — the RetrievalMode and
stage literal types below are re-declared locally rather than imported from
retrieval.search or evals.schema, which define the same shapes for their
own reasons.
opentelemetry-api is a base dependency (ADR-0022 decision 1), so it is
imported unconditionally — there is no guarded import and no hand-rolled
no-op tracer shim to keep in signature parity by hand. With no
opentelemetry-sdk installed and nothing configured, get_tracer() still
returns a tracer, but every span it creates is non-recording: no export, no
collector, no error. That's documented opentelemetry-api behavior, not a
groundkit fallback, and it's why this module is excluded from the
core-subset coverage gate on the same test that keeps runtime.py inside
it (see CONTRIBUTING.md): whether a module's absence changes what a
response contains. A tracer's absence never does — it changes only what
gets shipped to a collector alongside the response, so this module is
covered by the whole-package 80% floor instead, the same treatment
service/ gets for the same reason.
The otel extra (not installed by default) is what turns a span into
something a collector actually receives; see
ADR-0022
for the dependency shape and the span attribute allowlist that keeps query
text, source paths and document content out of every exported span.
groundkit.telemetry ¶
OpenTelemetry spans and structured JSON logging (ADR-0022).
This is a LEAF module: it imports nothing from groundkit.
retrieval/search.py imports this module to instrument its search
span, so an import in the other direction would be circular. The
RetrievalMode and Stage literal types below are therefore
re-declared locally rather than imported from retrieval.search or
evals.schema, which define the same shapes for their own reasons.
evals/metrics.py is this repo's existing precedent for a deliberate
leaf: no groundkit imports, so its tests construct plain values and this
module's tests construct plain values too.
opentelemetry-api is a base dependency (ADR-0022 decision 1), so it
is imported unconditionally below. There is no guarded import, no
try/except ImportError, and no hand-rolled no-op tracer shim — the
ADR rejects that shape explicitly: a hand-maintained fake has to be kept in
signature parity with the real tracer by hand, which is exactly the defect
class tests/test_protocol_conformance.py exists to catch elsewhere in
this codebase. With no opentelemetry-sdk installed and nothing
configured, get_tracer() returns a tracer whose spans are
non-recording: no export, no collector, no configuration, no error. That is
documented opentelemetry-api behaviour, not a groundkit fallback, and
SPEC.md §2's fail-closed rule does not apply to it — fail-closed governs
things whose absence changes an answer, and a tracer's absence changes no
answer (ADR-0022 decision 1).
The span attribute allowlist (ADR-0022 decision 3)¶
:func:span_attributes is the only way an instrumentation site may
attach data to a span. It is keyword-only and declares no **kwargs, so
an attribute outside the allowlist is a type error caught by mypy and by
the call site's own test suite, not something that depends on a reviewer
noticing it. That shape mirrors ADR-0001 hazard 3: a **kwargs that
silently absorbs whatever it is handed is a defect class this repo has
already been bitten by, and the fix here is structural rather than a
convention to remember.
Permitted, and nothing else: collection name, retrieval mode, retrieval
stage, top_k, result/candidate/chunk/document counts, duration, a typed
failure code, request id, HTTP status, the embedding identity triple
ADR-0004 defines (provider, model name, dimensions), and the chat
provider/model pair decision 5 permits on the synthesize span
specifically. Every one of those is either a bounded enum, a count, a
timing, or configuration the operator chose — never a value a user or a
document supplied, which is the line the allowlist is actually drawing.
Forbidden on every span, at every level — not just at INFO, because
a span has no levels and every attribute on a recording span leaves the
process: query text, chunk or document content, citation spans, absolute
source paths, and the values inside a metadata_filter (its keys would
be fine; its values are exactly where a tenant id or a customer name would
appear). There is no parameter for any of these on :func:span_attributes
and there must never be one — see tests/test_telemetry.py for the tests
that hold this allowlist to that promise structurally rather than by
review.
Attribute key naming scheme¶
Keys are dotted, OTel-style, under a groundkit. prefix. Concepts
specific to how a retrieval query was shaped nest under
groundkit.retrieval.*; the embedding identity triple nests under
groundkit.embedding.*; everything else — counts, duration, failure
kind, request id, HTTP status — is shared across the ingest/retrieve/
synthesize span sites and stays flat under groundkit.*:
collection->groundkit.collectionretrieval_mode->groundkit.retrieval.modestage->groundkit.retrieval.stagetop_k->groundkit.retrieval.top_kresult_count->groundkit.result_countcandidate_count->groundkit.candidate_countchunk_count->groundkit.chunk_countdocument_count->groundkit.document_countduration_ms->groundkit.duration_msfailure_kind->groundkit.failure_kindrequest_id->groundkit.request_idhttp_status->groundkit.http.status_codeembedding_provider->groundkit.embedding.providerembedding_model->groundkit.embedding.modelembedding_dimensions->groundkit.embedding.dimensionschat_provider->groundkit.chat.providerchat_model->groundkit.chat.model
Structured JSON logging¶
There is no central logging configuration in this repo today — the CLI
never calls logging.basicConfig. :func:configure_logging is the new
entry point that makes GROUNDKIT_LOG_FORMAT mean anything, and it is
deliberately a thin logging.Formatter subclass plus a handler
attachment rather than a dependency on structlog or loguru
(ADR-0022's alternatives section): every module already logs through
stdlib logging, so a formatter subclass gets the same JSON with no new
dependency and no repo-wide rewrite.
:class:JsonLogFormatter does not change any existing log message string
(ADR-0022 decision 4) — call sites keep their key=value message text
and gain structured fields via extra={...} alongside it, so assertions
written against today's message strings keep meaning what they meant.
JsonLogFormatter ¶
Bases: Formatter
Render one JSON object per log record.
The object always carries timestamp (ISO 8601, UTC — machine-
parseable across a collector's timezone rather than the host's local
one), level, logger, and message (the same string the
record's msg %% args would have produced under the human-readable
formatter — ADR-0022 decision 4 keeps existing message text unchanged).
Every field the call site passed via extra={...} is merged in
alongside those four; an exception field is added when the record
carries exception info.
A field from extra that collides with one of the four base keys is
dropped rather than allowed to overwrite it — an accidental
extra={"level": ...} must not be able to corrupt the record's actual
severity. Non-JSON-serializable values are coerced with str()
(json.dumps(..., default=str)) rather than raising, so a bad value
in extra can never crash the logging call site that produced it.
format ¶
Render record as a single-line JSON string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
record
|
LogRecord
|
The log record being emitted. |
required |
Returns:
| Type | Description |
|---|---|
str
|
A JSON-encoded object; see the class docstring for its shape. |
Source code in src/groundkit/telemetry.py
configure_logging ¶
Attach one stderr log handler to the root logger, idempotently.
There is no central logging configuration in this repo today, so this
is the entry point that makes GROUNDKIT_LOG_FORMAT mean anything.
GROUNDKIT_LOG_FORMAT=json selects :class:JsonLogFormatter; any
other value, or unset, keeps human-readable formatting, which is the
default (ADR-0022 decision 4) — a local grk search that started
printing JSON to a developer's terminal would be a regression in the
tool's primary use, not an improvement.
The handler always writes to stderr, never stdout.
service/mcp_server.py's _force_logging_to_stderr exists because
ADR-0014 decision 5 makes stdout JSON-RPC-only on the stdio transport —
one stray log record on stdout corrupts the rest of that session. This
function does not fight that guard: it never attaches to stdout in the
first place, so if run_stdio later calls
_force_logging_to_stderr, that sweep finds nothing of this
function's to re-point.
Calling this twice does not attach a second handler — a handler with
:data:_HANDLER_NAME already present on the root logger is treated as
"already configured," and the call becomes a level update only. That
matters because nothing prevents configure_logging from being
called from more than one entry point (a CLI command and a service
startup path, say) without either one knowing the other already ran.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
level
|
int
|
The root logger's level. Applied on every call — including a repeat call that finds the handler already attached — so a caller can still raise or lower verbosity without first checking whether an earlier call already configured logging. |
INFO
|
Source code in src/groundkit/telemetry.py
configure_tracing ¶
Install an SDK tracer provider when the environment asks for one.
This function is why any span in this package is ever recorded.
opentelemetry-api on its own returns a ProxyTracerProvider whose
spans are non-recording, and — this is the part that is easy to get
wrong and was gotten wrong here first — installing
opentelemetry-sdk and setting the OTEL_* variables does not
change that. Those variables are read by
opentelemetry.sdk._configuration, which runs under the
opentelemetry-instrument launcher, not on import. Absent an explicit
set_tracer_provider call, a fully-installed, fully-configured
process still exports nothing, silently and with no error — a
verification against the compose stack found exactly that.
The SDK import here is guarded, and it is the only guarded
opentelemetry import in this package. That does not contradict ADR-0022
decision 1, which requires instrumentation sites to import
unconditionally and rejects a hand-rolled no-op tracer shim: the sites
still import only opentelemetry-api and still need no guard, and no
shim exists — when the SDK is absent the real opentelemetry-api
no-op path is what runs. A bootstrap that must tolerate an optional
extra being absent is the one place a guard genuinely belongs.
Configuration is entirely the standard OTEL_* environment (ADR-0022
decision 2) — Resource.create() reads OTEL_SERVICE_NAME and
OTEL_RESOURCE_ATTRIBUTES, and each exporter's own constructor reads
the matching endpoint variable. Nothing is passed explicitly, so there is
no second vocabulary and no precedence rule to invent for those, and a
typo in an endpoint produces the SDK's own error rather than a groundkit
ConfigurationError. The wire protocol is the one exception: an
OTLPSpanExporter reads its endpoint from the environment but not its
own transport — that is fixed by which class was imported, HTTP/protobuf
in the http module and gRPC in the grpc one — so this function
reads OTEL_EXPORTER_OTLP_*PROTOCOL itself and picks the class. A
version of this that always imported the HTTP class shipped first: it
applied cleanly against a grpc-configured collector and then failed
at every export call, because nothing here was reading the variable the
docs already claimed it did.
Calling this is optional and its absence changes no answer, so a missing SDK is a warning rather than a failure — SPEC.md §2's fail-closed rule governs things whose absence changes a result, and ADR-0022 decision 1 says so explicitly to stop this being cited as precedent either way. What is not softened is the export path: a configured endpoint that cannot be reached surfaces as the SDK's own export error.
Idempotent: a second call finds a real provider already registered and
returns without replacing it (OTel itself only warns and ignores a
second set_tracer_provider, which would otherwise make the outcome
depend on call order).
Source code in src/groundkit/telemetry.py
249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 | |
get_tracer ¶
Return the package-wide OpenTelemetry tracer.
Every instrumentation site (Indexer.index_source,
Indexer.index_directory, Retriever.search,
Synthesizer.synthesize) calls this rather than
opentelemetry.trace.get_tracer directly, so they all publish under
the same "groundkit" instrumentation scope.
With no opentelemetry-sdk installed and no exporter configured, the
returned tracer's spans are non-recording: span.is_recording() is
False, attribute calls are accepted and discarded, and nothing is
exported. That is the documented behaviour of opentelemetry-api
alone (ADR-0022 decision 1), not a groundkit fallback path, and it costs
the caller nothing to instrument unconditionally.
Returns:
| Type | Description |
|---|---|
Tracer
|
A |
Source code in src/groundkit/telemetry.py
span_attributes ¶
span_attributes(
*,
collection: str | None = None,
retrieval_mode: RetrievalMode | None = None,
stage: Stage | None = None,
top_k: int | None = None,
result_count: int | None = None,
candidate_count: int | None = None,
chunk_count: int | None = None,
document_count: int | None = None,
duration_ms: float | None = None,
failure_kind: str | None = None,
request_id: str | None = None,
http_status: int | None = None,
embedding_provider: str | None = None,
embedding_model: str | None = None,
embedding_dimensions: int | None = None,
chat_provider: str | None = None,
chat_model: str | None = None,
) -> dict[str, str | int | float]
Build a span attribute dict from the ADR-0022 allowlist, and nothing else.
Every permitted attribute is its own explicitly typed keyword parameter.
There is no **kwargs: an attribute outside this allowlist is a
TypeError at the call site (and, before that, a mypy error), rather
than something a reviewer has to notice is missing from a list. See the
module docstring for the full allowlist, the forbidden set it
deliberately excludes, and the key naming scheme applied below.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
collection
|
str | None
|
The collection name, when reachable at the call site. |
None
|
retrieval_mode
|
RetrievalMode | None
|
|
None
|
stage
|
Stage | None
|
|
None
|
top_k
|
int | None
|
The requested result cutoff. |
None
|
result_count
|
int | None
|
Number of results returned. |
None
|
candidate_count
|
int | None
|
Number of candidates considered before cutoff. |
None
|
chunk_count
|
int | None
|
Number of chunks involved (e.g. produced by ingest). |
None
|
document_count
|
int | None
|
Number of documents involved. |
None
|
duration_ms
|
float | None
|
Elapsed wall-clock time in milliseconds. |
None
|
failure_kind
|
str | None
|
A typed, bounded failure label, when the span
recorded a failure. ADR-0022 decision 3 names
:mod: |
None
|
request_id
|
str | None
|
The request id assigned at the service boundary. |
None
|
http_status
|
int | None
|
The HTTP status code returned for the request. |
None
|
embedding_provider
|
str | None
|
ADR-0004's embedding identity triple, part 1. |
None
|
embedding_model
|
str | None
|
ADR-0004's embedding identity triple, part 2. |
None
|
embedding_dimensions
|
int | None
|
ADR-0004's embedding identity triple, part 3. |
None
|
chat_provider
|
str | None
|
Chat provider backing a synthesis call. |
None
|
chat_model
|
str | None
|
Chat model identifier backing a synthesis call.
ADR-0022 decision 5 permits "model identity" on the
|
None
|
Returns:
| Type | Description |
|---|---|
dict[str, str | int | float]
|
A dict containing only the arguments that were not |
dict[str, str | int | float]
|
by the dotted |
dict[str, str | int | float]
|
docstring. OTel attribute values may not be |
dict[str, str | int | float]
|
argument is omitted from the result rather than included with a |
dict[str, str | int | float]
|
|
Source code in src/groundkit/telemetry.py
358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 | |